GH-36411: [C++][Python] Use meson-python for PyArrow build system #45854

WillAyd · 2025-03-18T18:47:01Z

Rationale for this change

This helps simplify the steps to build pyarrow by leveraging Meson, a build system strongly inspired by Python's syntax. In it's current form, it requires Arrow to be installed on the host system, but in the future we may even be able to have PyArrow build Arrow as a subproject, as needed

What changes are included in this PR?

This PR adds Meson configuration files to the Python code base within Arrow.

Are these changes tested?

Yes

Are there any user-facing changes?

We may want to deprecate the traditional setup.py way of building PyArrow alongside this.

GitHub Issue: [Python] Improve the pyarrow build system #36411

github-actions · 2025-03-18T18:47:26Z

⚠️ GitHub issue #36411 has been automatically assigned in GitHub to PR creator.

eli-schwartz · 2025-03-18T20:35:05Z

python/meson.build

Before doing this, include the following code:

# no-op placeholder arrow_dep = dependency('', required: false) if get_option('wrap_mode') != 'forcefallback' arrow_dep = dependency('arrow', 'Arrow', modules: ['Arrow::arrow_shared'], required: false) endif

And then shift the rest to look like this:

if not arrow_dep.found() cmake = import('cmake') # further lookups # ... arrow_dep = arrow_proj.dependency('arrow_shared') endif

What this does:

check build options for wrap_mode, which is a builtin meson option allowing you to choose whether you wish to resolve bundled dependencies or look for system dependencies. It defaults to finding system dependencies, but when users run meson with --wrap-mode=forcefallback they are asking to explicitly avoid system deps

first try to find an arrow dependency, using both names it might be available as:

"arrow" (pkgconfig)

"Arrow" (cmake, yes capitalization does matter), with modules: ensuring we pick up the correct cmake find_package() variable

if it is not available, required: false means we continue to import the cmake subproject as a fallback

It is possible to avoid doing if/else checks:

$ cat subprojects/arrow.wrap [wrap-file] directory = arrow method = cmake [provide] arrow = arrow_static_dep

However, using wrap files with method=cmake doesn't (currently) allow you to pass your add_cmake_defines. If you didn't need any defines, then you could simply do this:

arrow_dep = dependency('arrow', 'Arrow', modules: ['Arrow::arrow_shared'])

and you would not need any if/else, it would automatically build the cmake subproject if either:

wrap-mode=forcefallback

no system arrow was found

How does that wrap file work? The directory to the cpp source is in arrow/cpp whereas the wrap file itself will be located in arrow/python/subprojects - how would that resolve to the right directory?

Since you included a symlink anyway, I chose not to bother including any mechanism for downloading the wrap contents. Meson skips over that because the directory already exists with the correct content.

The key benefit of the wrap file is that it allows specifying in ini syntax:

the subproject should use the method=cmake automatically, when used via dependency()

the autogenerated arrow_static_dep (maybe this should be arrow_shared_dep instead?) will fulfill dependency('arrow')

Again, it's missing the necessary cmake defines so it may not be worth pursuing further.

eli-schwartz · 2025-03-18T22:46:09Z

python/pyproject.toml

You could still use setuptools-scm with meson if you want.

project( 'pyarrow', # ...., version: run_command('python3', '-m', 'setuptools_scm', '--force-write-version-files', check: true).stdout().strip(), )

WillAyd · 2025-03-20T00:59:53Z

@kou I have made some offline progress on this, but one of the things I am getting stuck on is how the pyarrow C++ modules are being compiled. From what I understand, the current build process will compile Cython modules first (at least lib.pyx) and from that auto-generate lib.h and lib_api.h headers that the pyarrow modules can then reference (?)

Assuming that understanding is correct, where in the process are lib.h and lib_api.h being generated? I found the CMake command that copies them from the source to the build folder, but I can't figure out where they come from in the first place. Any guidance would be appreciated.

kou · 2025-03-20T01:07:17Z

The following codes may be related:

arrow/cpp/cmake_modules/UseCython.cmake

Lines 120 to 126 in 5e9fce4

    
             if(${property_is_api}) 
        
               set(_generated_files "${output_file}" "${_name}.h" "${_name}_api.h") 
        
             elseif(${property_is_public}) 
        
               set(_generated_files "${output_file}" "${_name}.h") 
        
             else() 
        
               set(_generated_files "${output_file}") 
        
             endif()

arrow/python/CMakeLists.txt

Line 674 in 5e9fce4

set_source_files_properties(pyarrow/lib.pyx PROPERTIES CYTHON_API TRUE)

WillAyd · 2025-03-20T01:07:40Z

Ah nevermind I think I have figured it out. So it looks like Cython generates the header files in the build directory when compiling lib.pyx, so the idea is to copy those header files to a directory structure in the build directory that the sources can resolve to.

I'll have to think about the best way to accomplish that via Meson.

python/pyarrow/src/arrow/python/numpy_to_arrow.cc

WillAyd · 2025-11-21T21:07:41Z

@github-actions crossbow submit -g python

github-actions · 2025-11-21T21:10:21Z

Revision: 7ca8999

Submitted crossbow builds: ursacomputing/crossbow @ actions-fea1cc1589

Task	Status
example-python-minimal-build-fedora-conda
example-python-minimal-build-ubuntu-venv
test-conda-python-3.10
test-conda-python-3.10-hdfs-2.9.2
test-conda-python-3.10-hdfs-3.2.1
test-conda-python-3.10-pandas-1.3.4-numpy-1.21.2
test-conda-python-3.11
test-conda-python-3.11-dask-latest
test-conda-python-3.11-dask-upstream_devel
test-conda-python-3.11-hypothesis
test-conda-python-3.11-pandas-latest-numpy-latest
test-conda-python-3.11-spark-master
test-conda-python-3.12
test-conda-python-3.12-cpython-debug
test-conda-python-3.12-pandas-latest-numpy-1.26
test-conda-python-3.12-pandas-latest-numpy-latest
test-conda-python-3.13
test-conda-python-3.13-pandas-nightly-numpy-nightly
test-conda-python-3.13-pandas-upstream_devel-numpy-nightly
test-conda-python-3.14
test-conda-python-emscripten
test-cuda-python-ubuntu-22.04-cuda-11.7.1
test-debian-12-python-3-amd64
test-debian-12-python-3-i386
test-fedora-42-python-3
test-ubuntu-22.04-python-3
test-ubuntu-22.04-python-313-freethreading
test-ubuntu-24.04-python-3

WillAyd

Sorry for all the PR spam...took a few tries to get the Windows CI green.

I think this is in a workable place now and all is green. Curious if you have any thoughts @raulcd @kou @pitrou

WillAyd · 2025-11-21T22:35:01Z

.github/workflows/python.yml

      PYARROW_TEST_LARGE_MEMORY: ON
      # Current oldest supported version according to https://endoflife.date/macos
-      MACOSX_DEPLOYMENT_TARGET: 12.0
+      MACOSX_DEPLOYMENT_TARGET: "12.0"


I enclosed this in parentheses so that it gets evaluated as the string "12.0" and not the number; otherwise some functionality in Meson was failing to unpack the major/minor version

Somewhat tangentially it looks like the minimum supported version right now is 14.0 - worth updating?

WillAyd · 2025-11-21T22:39:54Z

python/meson.build

+        check: true,
+    ).stdout().strip(),
+    license: 'Apache-2.0',
+    #license_files: ['../LICENSE.txt'],


This is going to reflect the same limitation that is being worked around in #47141

As far as I am aware, Meson supports the inclusion of files like this, but meson-python has opted for a strict interpretation of PEP-639 that requires the LICENSE.txt (and similar files) to exist within the python source. See some upstream discussion at mesonbuild/meson#14387

Is it possible to add a build step that copies the license file(s) to the right place before packaging?

Sure - meson has the meson.add_dist_script() function that should help. I'll give that a closer look here

WillAyd · 2025-11-21T22:42:46Z

python/pyarrow/tests/util.py

    else:
        new_pythonpath = module_path
    env['PYTHONPATH'] = new_pythonpath
+    env['MACOSX_DEPLOYMENT_TARGET'] = "14.0"


The cython tests that were running setup.py in a subprocess and linking against the pyarrow-exposed arrow libraries were failing on macOS without this

WillAyd · 2025-11-21T22:50:52Z

ci/scripts/python_build.bat

+@REM by default, CMake installs .lib import libs to lib and .dll libs to bin
+@REM delvewheel requires these to be side-by-side to properly vendor
+@REM https://github.com/adang1345/delvewheel/issues/66
+copy %CMAKE_INSTALL_PREFIX%\lib\*.lib %CMAKE_INSTALL_PREFIX%\bin\


This is pretty hacky but I'm not sure of a way around it...open to any thoughts.

The crux of the issue is that the CMake module on main will copy the Arrow libraries from the system into the package. meson-python is more strict about what you can copy into the package, and I don't believe it allows you to do that, pushing you instead to use tools like delvewheel (see comprehensive shared library documentation at https://mesonbuild.com/meson-python/how-to-guides/shared-libraries.html)

Using delvewheel gets the job mostly done, excepting the fact that the .lib and .dll files are not in the same location on the system, which delvewheel expects (see an upstream issue at adang1345/delvewheel#66)

I tried adding a CMake option of -DCMAKE_INSTALL_LIBDIR=bin to place the .lib files near the .dll files, but that broke the CMake package finding capabilities. I tried to then do -DCMAKE_INSTALL_RUNTIMEDIR=lib, but that also had unintended side-effects (and required patching of the Arrow CMake module, which hardcodes that to bin in the ARROW_ADD_LIB function)

This hack was the only way I could get this to work.

There's also the question of why the Windows CI job requires this in the first place, which I think has to do with the python_build.bat script installing Arrow C++ with the VSenv toolchain, while the python_test.bat script runs in the standard environment that appears to have the MinGW toolchain active.

The crux of the issue is that the CMake module on main will copy the Arrow libraries from the system into the package. meson-python is more strict about what you can copy into the package, and I don't believe it allows you to do that, pushing you instead to use tools like delvewheel

That's gonna be annoying for development. What about editable builds? (pip install -e ...)

I couldn't reproduce this locally on a Windows machine, so development seemed fine. I'll see if I can better pinpoint the root cause from CI

At least on Unix platforms, editable builds will still have any relevant Meson (not meson-python) rpath entries pointing to private copies of dependency libraries.

Windows doesn't have rpath because the technology is too powerful for them. :(

While the lack of RPATH support on Windows is unfortunate, I don't think that's relevant here (?). This is installing the package on the same machine that was used to build it, so I would expect it can still resolve to the system libraries, no?

@pitrou aha, that is very nice. That means it should work consistently across windows / Linux / macOS for libraries linked as conda dependencies (though admittedly it doesn't help much for arbitrary windows users via the official python redistributable who also want to use editable builds, but at a certain point I suppose you just have to give up and say that people doing developer builds need to use the tooling designed to make that work well on Windows).

Hmm I was mistakenly assuming that conda was being used for this Windows job, but on closer inspection that doesn't appear to be the case. It looks like the install prefix is being set to cygpath --absolute --windows /usr, so perhaps that is the issue

I don't know why the comment mentions delvewheel, though. We already use delvewheel successfully in the Windows wheel builds, FTR.

I see that this repo limits use of non-verified marketplace actions like conda-incubator/setup-miniconda and mamba-org/setup-micromamba. At the same time, I don't see any pre-built Windows docker images that already have conda.

Without knowing yet how Archery works, do you think its best to go down the marketplace action path or to set up something with docker/archery?

Hmm, I think installing mamba or equivalent manually may be the best course of action.

WillAyd · 2025-11-21T22:52:52Z

python/pyarrow/meson.build

+    endif
+endif
+
+gnu_symbol_visibility = host_machine.system() == 'darwin' ? 'default' : 'inlineshidden'


On my local box using inlineshidden reduces the overall size of shared libraries by 5%. On macOS it was segfaulting, which from cursory research has to do with how the PyMod_INIT function is exposed

WillAyd · 2025-11-21T23:41:58Z

@github-actions crossbow submit -g python

github-actions · 2025-11-21T23:44:58Z

Revision: 91acf11

Submitted crossbow builds: ursacomputing/crossbow @ actions-ab25d09306

Task	Status
example-python-minimal-build-fedora-conda
example-python-minimal-build-ubuntu-venv
test-conda-python-3.10
test-conda-python-3.10-hdfs-2.9.2
test-conda-python-3.10-hdfs-3.2.1
test-conda-python-3.10-pandas-1.3.4-numpy-1.21.2
test-conda-python-3.11
test-conda-python-3.11-dask-latest
test-conda-python-3.11-dask-upstream_devel
test-conda-python-3.11-hypothesis
test-conda-python-3.11-pandas-latest-numpy-latest
test-conda-python-3.11-spark-master
test-conda-python-3.12
test-conda-python-3.12-cpython-debug
test-conda-python-3.12-pandas-latest-numpy-1.26
test-conda-python-3.12-pandas-latest-numpy-latest
test-conda-python-3.13
test-conda-python-3.13-pandas-nightly-numpy-nightly
test-conda-python-3.13-pandas-upstream_devel-numpy-nightly
test-conda-python-3.14
test-conda-python-emscripten
test-cuda-python-ubuntu-22.04-cuda-11.7.1
test-debian-12-python-3-amd64
test-debian-12-python-3-i386
test-fedora-42-python-3
test-ubuntu-22.04-python-3
test-ubuntu-22.04-python-313-freethreading
test-ubuntu-24.04-python-3

github-actions bot added Component: Python awaiting review Awaiting review labels Mar 18, 2025

WillAyd mentioned this pull request Mar 18, 2025

[Python] Improve the pyarrow build system #36411

Open

eli-schwartz reviewed Mar 18, 2025

View reviewed changes

WillAyd force-pushed the use-meson-python branch 2 times, most recently from cf5b610 to b902e1d Compare March 18, 2025 21:50

github-actions bot added awaiting committer review Awaiting committer review and removed awaiting review Awaiting review labels Mar 18, 2025

WillAyd force-pushed the use-meson-python branch 6 times, most recently from eabf11f to 7be3f7b Compare March 18, 2025 22:43

eli-schwartz reviewed Mar 18, 2025

View reviewed changes

WillAyd force-pushed the use-meson-python branch 2 times, most recently from 05ff60d to 6e0a5fe Compare March 19, 2025 03:58

WillAyd force-pushed the use-meson-python branch 2 times, most recently from d67b903 to ba8b276 Compare March 20, 2025 18:01

WillAyd commented Mar 20, 2025

View reviewed changes

python/pyarrow/src/arrow/python/numpy_to_arrow.cc Outdated Show resolved Hide resolved

WillAyd force-pushed the use-meson-python branch 6 times, most recently from a2d07ad to 4ff818e Compare March 21, 2025 00:39

WillAyd force-pushed the use-meson-python branch 11 times, most recently from 5233199 to f266c86 Compare November 21, 2025 16:26

apacheGH-36411: [C++][Python] Use meson-python for PyArrow build system

aa8c6ce

WillAyd force-pushed the use-meson-python branch from f266c86 to aa8c6ce Compare November 21, 2025 17:32

WillAyd added 2 commits November 21, 2025 12:57

Non-meson configuration cleanups

7c66628

Further cleanups

3ab6724

github-actions bot removed the Component: C++ label Nov 21, 2025

Enable all Windows options

7ca8999

WillAyd force-pushed the use-meson-python branch from 6f72a9a to 7ca8999 Compare November 21, 2025 19:29

WillAyd commented Nov 21, 2025

View reviewed changes

Revert unnecessary test change

91acf11

WillAyd force-pushed the use-meson-python branch 4 times, most recently from bbe3c79 to cd7b67c Compare December 1, 2025 16:02

Try Windows using conda

6325d72

WillAyd force-pushed the use-meson-python branch from cd7b67c to 6325d72 Compare December 1, 2025 16:03

GH-36411: [C++][Python] Use meson-python for PyArrow build system #45854

Are you sure you want to change the base?

GH-36411: [C++][Python] Use meson-python for PyArrow build system #45854

Conversation

WillAyd commented Mar 18, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Rationale for this change

What changes are included in this PR?

Are these changes tested?

Are there any user-facing changes?

Uh oh!

github-actions bot commented Mar 18, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

eli-schwartz Mar 18, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

eli-schwartz Mar 18, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

WillAyd commented Mar 20, 2025

Uh oh!

kou commented Mar 20, 2025

Uh oh!

WillAyd commented Mar 20, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

WillAyd commented Nov 21, 2025

Uh oh!

github-actions bot commented Nov 21, 2025

Uh oh!

WillAyd left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

pitrou Nov 25, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

pitrou Nov 27, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

WillAyd commented Mar 18, 2025 •

edited

Loading

eli-schwartz Mar 18, 2025 •

edited

Loading

eli-schwartz Mar 18, 2025 •

edited

Loading

WillAyd commented Mar 20, 2025 •

edited

Loading

pitrou Nov 25, 2025 •

edited

Loading

pitrou Nov 27, 2025 •

edited

Loading