Skip to content

[build] Add a Bazel build: shared module, standalone CppInterOp pin and wheel-layout parity - #55

Open
conrade-ctc wants to merge 1 commit into
compiler-research:mainfrom
chicagotrading:bazel-build
Open

[build] Add a Bazel build: shared module, standalone CppInterOp pin and wheel-layout parity#55
conrade-ctc wants to merge 1 commit into
compiler-research:mainfrom
chicagotrading:bazel-build

Conversation

@conrade-ctc

@conrade-ctc conrade-ctc commented Aug 31, 2026

Copy link
Copy Markdown

CppInterOp's Bazel build depends on a shared module, cppyy_bazel, that lives
in compiler-research/cppyy at cppyy/bazel/. That repo is archived and the
four Python repos merged into this one, so the module has no home in the
surviving stack. CppInterOp's bazel (best-effort) job still clones cppyy to
find it, and both bazel build //... and bazel test //:tests fail there
today.

This PR gives the module a home here and adds a Bazel build for cppjit.

What it contains

  • bazel/ — the shared module, moved out of the archived cppyy repo and
    renamed cppjit_bazel. It holds the copts and linkopts, the @llvm module
    extension, the wheel-layout staging rule and the test macros. It is its own
    Bazel module, so cppjit and CppInterOp can both depend on it without a
    bzlmod cycle.
  • A root Bazel build. One merged cc_shared_library, mirroring CMake:
    every src/cpyrt/*.cxx plus the two interop translation units link into
    python/cppjit/libcppjit.so. stage_files recreates the wheel layout
    beside it, which is what cppinterop_paths() anchors at its own dladdr
    location.
  • A standalone CppInterOp pin. The build fetches CppInterOp from the
    commit CMakeLists.txt pins, so a fresh clone needs no sibling checkout.
    Only LLVM_DIR comes from the host, as with CMake. A commented
    local_path_override keeps the sibling dev loop available — the Bazel
    equivalent of CMake's CPPINTEROP_SOURCE_DIR.
  • ${ORIGIN} expansion in interpreter args. No installer knows its
    absolute prefix at build time. libcppjit.so now rewrites
    CPPINTEROP_EXTRA_INTERPRETER_ARGS, so a consumer passes the token through
    verbatim.
  • The best-effort Bazel CI job, restored from the archived cppyy repo.
    Non-gating (continue-on-error); the standalone pin removes the sibling
    clones the old job needed.
  • extra_tags on cppinterop_cc_test. A consumer can select or exclude
    one test with --test_tag_filters.

Standalone constraint

The build must work from a fresh clone with nothing beside it. Verified: this
branch cloned into an empty directory, no sibling checkout, a fresh
--output_base, and --nosystem_rc --nohome_rc so no host bazelrc can
supply a registry.

LLVM_DIR=/path/to/llvm bazelisk test //...
Executed 26 out of 26 tests: 26 tests pass.

//:test_selflocation is the layout gate: it imports and JITs with cwd /
and no path environment variables, so a regression in the dladdr-relative
resource lookup fails here instead of only in an installed wheel.

Every commit in the series builds on its own.

CppInterOp pin

Pinned at 9802d619, the same commit CPPINTEROP_GIT_TAG names, so the Bazel
and CMake builds compile the same backend sources. The in-tree patch only
renames the shared module in the fetched tree; it goes away once the pinned
commit carries the rename.

Parity with pip install .

//:site_packages reproduces the pip install . payload class for class —
one unversioned libclangCppInterOp.so, the CppInterOp headers, the clang
builtin resource dir, libcppjit.so with a matching soname, and the
pure-Python sources — with a single remaining difference: Bazel ships the
library unstripped (181 MB against 156 MB stripped). .dynsym survives
stripping, so the dlsym-based dispatch resolves either way. Happy to add a
strip step if you want exact bytes.

Follow-up

A companion PR in CppInterOp repoints its bazel_dep and
local_path_override at cppjit_bazel, updates the CI clone, wires the test
sources CMake already lists, and tags VTableOverlayCrossTUBench as a
benchmark. It needs this PR first.

@conrade-ctc

Copy link
Copy Markdown
Author

@aaronj0, here's the bazel infra added back in for cppjit. I'll open another PR against CppInterOp to adjust properly once this is in, and we should be cleanly bazel capable again!

Comment thread .github/workflows/bazel.yml Outdated
Comment thread requirements_bazel.txt Outdated
Comment thread MODULE.bazel Outdated
Move the shared cppjit_bazel module out of the archived cppyy repo, add
a root Bazel build that mirrors the CMake/wheel layout, and restore the
best-effort CI job.

- bazel/: the shared module (copts, the @llvm extension, the wheel-layout
  staging rule, the test macros) as its own Bazel module.
- Root build: one merged cc_shared_library links into
  python/cppjit/libcppjit.so; stage_files recreates the wheel layout;
  test_selflocation gates the dladdr-relative resource lookup.
- CppInterOp comes from the commit CMakeLists.txt pins, so a fresh clone
  needs no sibling checkout. Only LLVM_DIR comes from the host.
- libcppjit.so expands the ${ORIGIN} token in
  CPPINTEROP_EXTRA_INTERPRETER_ARGS.
- cppinterop_cc_test takes extra_tags for --test_tag_filters.
- test/test_main.py imports the cppjit_bazel path fixups. The py_test
  bootstrap builds sys.path in-process, so site cannot auto-import them.
- ci: the best-effort, non-gating bazel workflow, restored from the
  archived repo without the sibling clones.

Co-developed-with-the-help-of: Claude Code (Fable 5, human in the loop)
@conrade-ctc

conrade-ctc commented Aug 31, 2026

Copy link
Copy Markdown
Author

@mcbarton / @aaronj0, all updated now! thanks.

@conrade-ctc

Copy link
Copy Markdown
Author

One more thing the 3.14 bump surfaced: a latent bug. bazel/sitecustomize.py repoints sysconfig's INCLUDEPY — python-build-standalone bakes it as /install/include/pythonX.Y, which does not exist at runtime — and anchors the runfiles-relative LD_LIBRARY_PATH segments to absolute paths. It relied on site auto-importing the magic module name. Under rules_python 1.8.5 the py_test bootstrap builds sys.path in-process, after site runs, so the module was never imported and both fixups were silently inert. test_regression catches the first; the second would only have shown up as a dlopen failure on GitHub Actions. test/test_main.py now imports the module explicitly, guarded by try/except ImportError so the pip/CMake path is unaffected. 26/26 green on 3.14.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants