macOS wheels: arm64 CI¶
Notes captured during the 0.35.0.x release-pipeline hardening (PR #98), recorded so future maintainers don't re-diagnose them.
The macOS build runs on macos-15 (arm64; Intel macOS wheels were dropped for 0.36), and
have sharp edges
the Linux build doesn't. The dangerous part: each one fragments a release
rather than failing a build, so it surfaces as a gappy wheel set on PyPI
after the tag is cut — not as CI red you'd catch in review.
upload-artifact times out ~⅓ of the time on arm64 runners¶
actions/upload-artifact CreateArtifact: Request timeout is a known, unfixed arm64-runner bug
Up to 0.35.0.x, on macos-14 (and any arm64 macOS runner), actions/upload-artifact@v4's archive wheel step failed
roughly one upload in three with
Failed to CreateArtifact: Failed to make request after 5 attempts: Request timeout.
The wheel built fine — the failure is purely the upload handshake to GitHub's
artifact service. It is a long-standing open defect
(actions/upload-artifact#569,
#527); the Actions
team acknowledged it in Dec 2024 and it is still open. The action's own 5
internal retries exhaust within the same bad network window, so they don't
help — a fresh outer re-invocation usually succeeds.
Why it fragmented releases: in the 0.35 per-platform macOS workflow publish had
needs: [build], so one failed upload marked the whole build matrix failed
and the entire macOS publish was skipped — zero macOS wheels for that release. At
~⅓ per job across 4 Python builds, P(all 4 upload) = (⅔)⁴ ≈ 20%, so ~80% of
macOS releases would otherwise have needed a manual job re-run. In the current
single wheels.yml, publish needs every build job, so a failed macOS upload
blocks the release on every platform.
Fix (wheels.yml): retry the archive wheels step up to 3× with
overwrite: true on the retries (clears any half-created artifact). The
final attempt has no continue-on-error, so a genuine outage still
fails loud — never a silently-missing wheel.
The runner label is an architecture, not just an OS version¶
Lowering runs-on: macos-15 silently flips the wheel from arm64 to x86_64
GitHub's macOS runner labels split by architecture, not just OS age:
| label | arch | notes |
|---|---|---|
macos-12 |
Intel x86_64 | removed Dec 2024 |
macos-13 |
Intel x86_64 | deprecation path |
macos-14 |
arm64 | what we built on up to 0.35.0.x |
macos-15 |
arm64 | what we build arm64 on |
macos-15-intel |
Intel x86_64 | built x86_64 for 0.36.0.0rc1 only |
We ship macosx_15_0_arm64 (up to 0.35.0.x, macosx_14_0_arm64). "Lowering the macOS version" to macos-13/-12
to dodge a runner problem (e.g. the upload flakiness above) does not keep
the same wheel — it switches the build to x86_64, orphaning every
Apple-Silicon user. And it wouldn't even help: the upload bug reproduces on
macos-13-xl-arm64 too, so it is arm64-runner-inherent, not OS-version
specific.
The legitimate "lower the macOS version" lever is
MACOSX_DEPLOYMENT_TARGET (a build-env var, independent of the runner) —
it sets the wheel's minimum macOS (macosx_13_0_arm64, …) for broader
install compatibility without touching the runner or the architecture. Use
that, not the runner label. scripts/native_stack.py declares it
(OSX_DEPLOYMENT_TARGET, 12.0) and exports it to colcon and the bindings' build;
the repair tags the wheel with it and refuses any bundled library built for a
newer macOS, which includes the current Homebrew libomp (15.0, see below), so
the workflow bundles the pixi env's conda-forge libomp (11.0).
STABLE_ABI: the wheel under test is not the Python under test¶
3.13–3.15 are verified by installing the cp312-abi3 wheel, not by building a 3.13–3.15 wheel
nanobind's STABLE_ABI build emits a single cp312-abi3 wheel that pip
installs on CPython 3.12, 3.13, 3.14+ — on every platform. So 3.13–3.15
support is real on macOS and Windows even though no 3.13–3.15 wheel is
built. To verify it in CI, each build job installs the wheel it just built
into a fresh venv per interpreter through the test_python_ver action
(.github/actions/test_python_ver), once per Python:
- name: test Python 3.13
uses: ./tesseract_nanobind/.github/actions/test_python_ver
with:
python_version: "3.13"
wheel_dir: ${{ github.workspace }}/tesseract_nanobind/test-wheels
The build jobs run it for 3.13, 3.14 and 3.15 on pushes, tags and scheduled
runs, and test-artifacts runs it for 3.12 on a fresh runner, the only suite
run a pull request gets. Every leg
blocks. Before PR #98 only Linux did this; macOS and
Windows tested 3.9 + 3.12 only, leaving the cross-platform abi3 claim
asserted but not verified. Now all three platforms verify it.
Sharing OpenMP with conda and pixi consumers¶
scripts/repair_macos_wheel.py bundles all dependencies into .dylibs/ and
leaves OpenMP to the dynamic linker: OpenMP load commands become
@rpath/libomp.dylib, with the installed Python environment's lib/ first,
then the bundled .dylibs/ directory. Changed Mach-O images are re-signed, and
the script rewrites the wheel's RECORD metadata.
scripts/check_wheel_openmp.py checks both import orders and the actual dyld
image paths. Plain consumers must load exactly one bundled runtime; the
test-wheel-openmp CI job must load exactly the environment runtime and pass
Descartes planning tests. Both this job and fresh-runner artifact tests gate
publishing. In the --lib-dir mode the wheel workflow uses, the repair leaves
the plugin YAML unchanged.
The current Homebrew runtime is built for macOS 15.0, above the 12.0 deployment
target, so the repair refuses a wheel that bundles it
(DeploymentTargetExceededError). The pixi env's conda-forge libomp
(llvm-openmp) is built for 11.0 and passes, so both builds take it from there:
the local pixi tasks pass $CONDA_PREFIX as --libomp-prefix, and the
workflow's macOS leg installs the default env with setup-pixi (locked:
true) and passes the same env's $CONDA_PREFIX. Its version is the one
pixi.lock pins; the native-cache key records it (scripts/wheel_cache.py).
Why the bundled-only wheel fails under conda (#136)
Wheels up to 0.35.0.8 were repaired by delocate-wheel, which bundled libomp and
pointed everything at that copy.
Install that wheel into a conda or pixi environment and the process gets
two. MEASURED on macOS 14 arm64, binding 0.35.0.7, in an environment whose
numpy comes from conda-forge:
| Image | Provenance | Loaded by |
|---|---|---|
<env>/lib/libomp.dylib |
conda-forge llvm-openmp |
numpy, at import — and it initialises the runtime |
<env>/lib/python3.13/site-packages/tesseract_robotics/.dylibs/libomp.dylib |
the delocated wheel | import tesseract_robotics, mapped but idle |
Different builds, different SHA-256. They coexist quietly, because the wheel's copy is only mapped — nothing initialises it until some plugin opens a parallel region. Descartes' ladder solver is the first thing that does, which is why an entire motion test suite can pass while this is sitting there:
- no flag:
OMP: Error #15: Initializing libomp.dylib, but found libomp.dylib already initialized, and the process aborts before a single waypoint is planned; KMP_DUPLICATE_LIB_OK=TRUE: survivesnum_threads=1, and segfaults atnum_threads=2and3(exit 139, 84-waypoint contour onabb_irb2400). The flag lets the second runtime load; it does not make running on both safe, exactly as its own warning says.
The control confirms the cause. Point the wheel's .dylibs/libomp.dylib
at the environment's copy, so that dyld loads one runtime, and the same
solve runs at 1, 2, 4 and 8 threads without the flag.
The fix doesn't, on its own, make Descartes faster: with vertex collision on, a profile puts ~97 % of a solve in serial setup — one contact-manager clone per waypoint, see the Descartes section of the planning guide — so the threads it restores mostly wait.
A Python exception crossing two extension modules lost its type¶
Every module that defines a trampoline calls register_python_error_translator()
A Python override that raises reaches C++ as nanobind::python_error. The
extensions are built NB_STATIC, so each .so holds its own strong
typeinfo for nanobind::python_error (nm -m lists one per module), and
macOS libc++ compares default-visibility typeinfo by pointer. When the
trampoline is in one module and the bound function that called it in
another (ConstraintSet in _trajopt_ifopt, called by
TrajOptQPProblem / TrustRegionSQPSolver in _trajopt_sqp), the
caller's catch (python_error &) misses, and nanobind's default
translator turns the error into RuntimeError(what()): an IndexError
arrived as RuntimeError with the traceback as its message
(#156).
A call that stays inside one module kept its type.
register_python_error_translator() (src/tesseract_nb.h) registers a
translator that runs the defining module's code, so its catch tests that
module's typeinfo and restores the Python exception. It must be static:
as an inline function its lambda was a weak symbol, dyld coalesced it
across images, and every module registered the first-loaded module's
copy, which fixed nothing. Linux (libstdc++) and Windows (MSVC) compare
typeinfo by name, so they should not lose the type (not measured; the
regression test tests/trajopt_sqp/test_python_set_exception_type.py
runs on all three). The alternative, one shared libnanobind instead of
NB_STATIC, would give the process a single typeinfo, at the cost of
another shared library in all three wheel layouts.