Dependency TODOs: enablements & clean-ups

[h-vetinari]

From looking at the list of third-party deps, we could still do better with:

• azure bindings
  • unix (lib): Add Dependencies for Arrow's Azure Filesystem #1431
  • unix (py): Build PyArrow with AzureFS Support pyarrow-feedstock#126
  • win (lib): Rebuild for Azure Core 1.16.3; drop CUDA 11.8; enable azure on windows #1919 (+ backports back to v21)
    • previously, Support azure bindings also on windows #1452 was blocked on [C++] Error when building Arrow-CPP w/ Azure Filesystem for Conda-Forge apache/arrow#41990 for a long time
  • win (py): add azure on win; rebuild for py3.14t; fix substrait; drop CUDA 11.8 pyarrow-feedstock#169 (+ backports back to v21)
• jemalloc (not possible?)
• gcp: supported for a long time, but version updates currently stuck
  • someone needs to go fix google-cloud-cpp-core v2.43.0 google-cloud-cpp-core-feedstock#46 etc.
• opentelemetry
  • Build shared libs, add run-export, register in global pinning cpp-opentelemetry-sdk-feedstock#38
  • unix: Build with opentelemetry support (on unix) #1048
  • win: Enable opentelemetry support on windows #1924
    • blocked on [C++] ENH: enable building against system opentelemetry on windows apache/arrow#45033
• orc for windows
  • Build for windows orc-feedstock#54
  • enabled for libarrow (as of db46b2c)
  • [Python] BUG: Reading ORC segfaults on windows (if TZDIR isn't set) apache/arrow#36026
  • ORC-1684: [C++] Find tzdb without TZDIR when in conda-environments apache/orc#1882
  • [15.x] Rebuild for orc 2.0.1; enable orc-on-windows also for pyarrow #1086
  • don't fail tzdb availability check prematurely #1424
  • enable orc-on-windows also for pyarrow pyarrow-feedstock#122
  • GH-41755: [C++][ORC] Ensure setting detected ORC version apache/arrow#41767
• nlohmann_json (as of c548fc2)
• substrait (not yet in conda-forge)
• ucx (as of c548fc2)

Switch to shared builds for (if/once available):

• google-cloud-cpp (win; not yet available in cf)
• gflags (win; available, but apparently symbols aren't needed; clarified as of bda7682)
• glog (win; available, but apparently symbols aren't needed; clarified as of bda7682)
• grpc (win; available as of 1.55, which requires protobuf >=4.23; fixed as of 025b77a)
• thrift (needed fixes for CMake metadata; fully cleaned up as of 2af3d60)

Bindings:

• skyhook (needs librados)
• tensorflow (though we should use libtensorflow, if at all, because it's a very heavy dependency to pull in)

[xhochy]

We cannot unvendor jemalloc as the build is with special options. Unverdoring it would have an impact on the default allocator.

[jakirkham]

Would it be possible to just build a variant of the jemalloc package with options needed?

[jakirkham]

Would not count on ucx being supported on other architectures OSes. Would also like to understand why that is relevant

[h-vetinari]

Would also like to understand why that is relevant

I have a maximalist approach to packaging the bindings offered by a given library -- if the library offers integration for it, why shouldn't we package it?¹

Users rely on conda-forge because a lot of these packages are really hard to build from source, so if a binding is not supported by us, it's effectively unusuable (even moreso if the distributed wheels also turn it off).

Footnotes

1. whether to break a package into separate outputs is a separate question ↩

[jakirkham]

Sorry I should clarify, what I mean is why are other OSes relevant? Asking since ucx likely will remain Linux only (possibly macOS at some point and basically no chance of Windows support)

[h-vetinari]

Sorry I should clarify, what I mean is why are other OSes relevant? Asking since ucx likely will remain Linux only (possibly macOS at some point and basically no chance of Windows support)

Not relevant indeed, unless they become: a) available for the respective platform & b) are supported by arrow on that platform.

[RAMitchell]

This may be more or a problem for upstream arrow, but this list of dependencies seems huge. Do all users need every cloud integration?

libarrow 19.0.1 hb3532e4_6_cuda
-------------------------------
file name   : libarrow-19.0.1-hb3532e4_6_cuda.conda
name        : libarrow
version     : 19.0.1
build       : hb3532e4_6_cuda
build number: 6
size        : 8.4 MB
license     : Apache-2.0
subdir      : linux-64
url         : https://conda.anaconda.org/conda-forge/linux-64/libarrow-19.0.1-hb3532e4_6_cuda.conda
md5         : d80e49f6afd3a03d5fc0faf980648b42
timestamp   : 2025-04-02 05:38:32 UTC
constraints : 
  - arrow-cpp <0.0a0
  - parquet-cpp <0.0a0
  - apache-arrow-proc =*=cuda
dependencies: 
  - __glibc >=2.17,<3.0.a0
  - aws-crt-cpp >=0.31.1,<0.31.2.0a0
  - aws-sdk-cpp >=1.11.510,<1.11.511.0a0
  - azure-core-cpp >=1.14.0,<1.14.1.0a0
  - azure-identity-cpp >=1.10.0,<1.10.1.0a0
  - azure-storage-blobs-cpp >=12.13.0,<12.13.1.0a0
  - azure-storage-files-datalake-cpp >=12.12.0,<12.12.1.0a0
  - bzip2 >=1.0.8,<2.0a0
  - glog >=0.7.1,<0.8.0a0
  - libabseil * cxx17*
  - libabseil >=20250127.1,<20250128.0a0
  - libbrotlidec >=1.1.0,<1.2.0a0
  - libbrotlienc >=1.1.0,<1.2.0a0
  - libgcc
  - libgcc-ng >=12
  - libgoogle-cloud >=2.36.0,<2.37.0a0
  - libgoogle-cloud-storage >=2.36.0,<2.37.0a0
  - libopentelemetry-cpp >=1.19.0,<1.20.0a0
  - libprotobuf >=5.29.3,<5.29.4.0a0
  - libre2-11 >=2024.7.2
  - libstdcxx
  - libstdcxx-ng >=12
  - libutf8proc >=2.10.0,<2.11.0a0
  - libzlib >=1.3.1,<2.0a0
  - lz4-c >=1.10.0,<1.11.0a0
  - orc >=2.1.1,<2.1.2.0a0
  - re2
  - snappy >=1.2.1,<1.3.0a0
  - zstd >=1.5.7,<1.6.0a0

[raulcd]

There are some efforts to try and move filesystems outside of libarrow on upstream Arrow but this is quite a big task with limited resources at the moment. Currently, when built, filesystems are part of the core libarrow and can't be split.
Some of the efforts already being done to achieve it:

• [C++] Allow filesystems to be implemented in separate library (+ move remote filesystems out of libarrow into their own shared libraries) apache/arrow#38309
• [C++] Move S3FileSystem to a separate module apache/arrow#40343
• [C++] Move LocalFileSystem to a separate module apache/arrow#40342

The vision is to have separate modules libarrow_s3.so, libarrow_gcs.so, etcetera. This is something currently on the project roadmap.

[h-vetinari]

@raulcd, is there a newer issue for this to track somewhere? AFAIU, the various cloud provider bindings have not moved out of libarrow yet.

[raulcd]

As far as I know there's currently no issue tracking the filesystems split work. The only slightly related issue open I can find is:

• [C++] Add key-value pairs to the FileSystemFactory interface apache/arrow#46369

[h-vetinari]

Thanks for the response! I think such an issue should be opened. Could you perhaps open one? I can do it as well if you'd like, but I'm probably not going to do the best job at reflecting all the previous efforts towards this.