diff --git a/README.md b/README.md index a63751a..88a182d 100644 --- a/README.md +++ b/README.md @@ -1,27 +1,220 @@ # CppJIT -CppJIT is the consolidated monorepo packaging of the compiler-research forks of the [cppyy](https://github.com/wlav/cppyy) package, a Python-C++ interoperability package based on LLVM, leveraging [CppInterOp](https://github.com/compiler-research/CppInterOp) for the Clang-REPL interpreter backend and Compiler-as-a-Service facilities. Star us and stay tuned for pip releases coming summer 2026. +[![CI](https://github.com/compiler-research/cppjit/actions/workflows/ci.yml/badge.svg)](https://github.com/compiler-research/cppjit/actions/workflows/ci.yml) +[![Nightlies](https://github.com/compiler-research/cppjit/actions/workflows/nightly.yml/badge.svg)](https://github.com/compiler-research/cppjit/actions/workflows/nightly.yml) +[![Wheels](https://github.com/compiler-research/cppjit/actions/workflows/wheels.yml/badge.svg)](https://github.com/compiler-research/cppjit/actions/workflows/wheels.yml) +[![Python](https://img.shields.io/badge/python-3.12%20%7C%203.13%20%7C%203.14-blue)](https://github.com/compiler-research/cppjit) +[![License](https://img.shields.io/badge/license-BSD--3--Clause--LBNL-green)](https://spdx.org/licenses/BSD-3-Clause-LBNL.html) -## Build requirements +**Bridge exploratory Python workflows with your production C++ codebases** -- LLVM/Clang 21 - - Installed system-wide (e.g. `apt install llvm-21-dev clang-21`) - - Installed via your favourite package manager (e.g. `conda install -c conda-forge "llvmdev=21" "clangdev=21"`) - - To use a source build of LLVM, pass the path to pip like `pip install . --config-settings=cmake.define.LLVM_DIR=/path/to/build/lib/cmake/llvm` -- Python 3.12+ with development headers (e.g. `apt install python3.14 python3.14-dev`) -- CMake 3.20+ +CppJIT embeds an interactive C++ JIT compiler in Python: write or import +C++ at runtime and use its functions, classes, and templates as if they +were Python. There is +no wrapper code to generate or maintain, no C++ build system to integrate, +and no FFI layer: bindings materialize automatically on demand. -### Standard installation: +Run-time binding generation enables: -Run `pip install .` +- **Detailed specialization** of each call at the point of use +- **Lazy loading** for reduced memory use in large scale projects +- **Python-side cross-inheritance and callbacks** for working with C++ + frameworks +- **Run-time template instantiation**, so the binding surface never has to + be enumerated ahead of time +- **Automatic object downcasting** and **exception mapping** +- **Interactive exploration** of C++ libraries from the Python prompt -### Development build with CMake: +CppJIT supports user-developed C++ frameworks and third-party C++ libraries +from standard package managers, and lets you leverage them from your Python +application or build a Python-based DSL layer on top. CppJIT is the successor of the +[cppyy](https://github.com/wlav/cppyy) project, rebuilt on +[CppInterOp](https://github.com/compiler-research/CppInterOp) and the +clang-repl C++ interpreter in LLVM/Clang. + +## How it works + +A CPython extension builds usable Python proxies for all C++ entities — +functions, classes, templates, variables — and the embedded JIT compiles +the C++ they touch, lazily: + +```python +import cppjit + +cppjit.cppdef(""" +#include +#include +double mean(const std::vector& xs) { + return std::accumulate(xs.begin(), xs.end(), 0.0) / xs.size(); +}""") + +cppjit.gbl.mean([1.0, 2.0, 3.0, 4.0]) # 2.5 — a Python list converts to std::vector +``` + +C++ entities come with Python ergonomics — constructors, operators, data +members: + +```python +cppjit.cppdef(""" +struct Vec2 { + double x, y; + Vec2 operator+(const Vec2& o) const { return {x + o.x, y + o.y}; } +};""") + +c = cppjit.gbl.Vec2(1, 2) + cppjit.gbl.Vec2(3, 4) +c.x, c.y # (4.0, 6.0) +``` + +CppInterOp enables this by providing the API for runtime reflection and +driving Clang and the underlying JIT infrastructure. + +Templates instantiate on demand, and STL containers behave like Python +containers: + +```python +cppjit.cppdef(""" +#include +template +T largest(const std::vector& xs) { return *std::max_element(xs.begin(), xs.end()); } +""") + +v = cppjit.gbl.std.vector['int']([3, 1, 4, 1, 5]) +cppjit.gbl.largest(v) # 5; largest is compiled at this call +len(v), list(v) # vectors support len(), iteration, indexing +``` + +Python callables pass into C++ as function pointers: + +```python +cppjit.cppdef(""" +template +R callme(R (*f)(U...), A &&...args) { + return f(args...); +}""") + +def callback(x: int, y: float) -> float: + return x + y + +cppjit.gbl.callme(callback, 123, 321.5) # 444.5 +``` + +NumPy arrays pass zero-copy; the C++ side works on the same buffer: + +```python +import numpy as np +a = np.arange(6, dtype=np.float64) + +cppjit.cppdef("void scale(double* xs, std::size_t n, double f) { while (n--) xs[n] *= f; }") +cppjit.gbl.scale(a, a.size, 10.0) +a # array([ 0., 10., 20., 30., 40., 50.]) — same buffer, no copy +``` + +An installed C++ library binds at run time, with no binding code written +for it: + +```python +import cppjit +cppjit.include('zlib.h') # bring in the C++ declarations +cppjit.load_library('libz') # load the symbols +cppjit.gbl.zlibVersion() # '1.3' — call the library directly +``` + +## Use cases + +- **Numerics and data science.** Move a hot loop into performant C++ in the same session. +- **Existing C++ codebases.** This technology originated in the field of high-energy + physics, where it binds very large existing C++ codebases for exploratory particle-physics + analysis. +- **Template-heavy APIs.** STL, Eigen, user templates: instantiations + happen lazily at the call site. +- CppInterOp also provides the C++ interop in + [jank](https://jank-lang.org) (Clojure on LLVM) and + [xeus-cpp](https://github.com/compiler-research/xeus-cpp) (interactive + C++ in Jupyter notebooks). + +## Source build requirements + +- LLVM/Clang development packages version 21 or 22 +- Python 3.12+ with development headers +- CMake 3.21+ and a C++20 compiler: g++ 13+, or a clang matching your LLVM + major (Ubuntu's stock clang-18 fails against LLVM 21/22 headers) +- Network access on the first build (CppInterOp is cloned during the build) + +## Install + +
+Ubuntu 24.04 + +```bash +sudo apt-get install -y git cmake make g++ python3-dev python3-venv python3-pip \ + wget lsb-release software-properties-common gnupg libzstd-dev libedit-dev +# stock apt ships LLVM 18; use apt.llvm.org. libzstd-dev and libpolly-21-dev +# are required or the build fails mid-way. +wget https://apt.llvm.org/llvm.sh && sudo bash llvm.sh 21 +sudo apt-get install -y llvm-21-dev libclang-21-dev clang-21 libpolly-21-dev + +python3 -m venv venv && source venv/bin/activate +git clone https://github.com/compiler-research/cppjit.git && cd cppjit +pip install -v . --config-settings=cmake.define.LLVM_DIR=/usr/lib/llvm-21/lib/cmake/llvm +``` + +
+ +
+macOS + +```bash +brew install llvm@21 cmake ninja +python3 -m venv venv && source venv/bin/activate +git clone https://github.com/compiler-research/cppjit.git && cd cppjit +pip install -v . --config-settings=cmake.define.LLVM_DIR="$(brew --prefix llvm@21)/lib/cmake/llvm" +``` + +
+ +### Check the install + +Run from any directory outside the checkout (the in-tree `python/cppjit` +shadows the installed extension): + +```bash +cd /tmp && python -c "import cppjit +cppjit.cppdef('int f(int x) { return x + 1; }') +print(cppjit.gbl.f(41))" +``` + +## Tests + +```bash +pip install -r requirements.txt +sudo apt-get install -y libboost-dev libeigen3-dev # optional; those tests skip without them +cd test +make -j4 # builds the *Dict.so dictionaries the tests load +python -m pytest -ra --tb=short +``` + +## Development builds + +Editable install with a persistent build directory; a one-file change +rebuilds incrementally: + +```bash +pip install scikit-build-core +pip install --no-build-isolation -ve . \ + --config-settings=build-dir=build \ + --config-settings=cmake.define.LLVM_DIR=$LLVM_DIR +``` + +To co-develop CppInterOp alongside cppjit, point the build at a local +checkout; it overrides the pinned tag: ```bash -mkdir build && cd build -cmake .. -DLLVM_DIR=/path/to/llvm/lib/cmake/llvm -DCMAKE_BUILD_TYPE=Debug -cmake --build . -j$(nproc) -cmake --install . --prefix /path/to/install +git clone https://github.com/compiler-research/CppInterOp.git ../CppInterOp +pip install --no-build-isolation -ve . \ + --config-settings=build-dir=build-local \ + --config-settings=cmake.define.LLVM_DIR=$LLVM_DIR \ + --config-settings=cmake.define.CPPINTEROP_SOURCE_DIR=$PWD/../CppInterOp ``` -To currently set up a source build, please look at the instructions at https://github.com/compiler-research/CppInterOp +Keep the checkout API-compatible with the pinned `CPPINTEROP_GIT_TAG` in +`CMakeLists.txt`.