CLI tool for generating C++ sources at CMake configure stage

[ahayzen-kdab]

Requirements

• Don't call cargo build at the configure stage of CMake cargo build is invoked at CMake configure step #106
• Don't generate files into target folder but into OUT_DIR or CMake build folder so that we don't collide and can do workspace sharing cargo: use workspace for shared building between cxx-qt packages #84
• Automatically output the sources from cxx-qt-lib
• C++ namespace prefix should be an option somewhere
• Add the generated sources to the CMake target

Solution

CLI Tool

As suggested in #106 we should use a CLI tool similar to cxxbridge ( https://github.com/dtolnay/cxx#non-cargo-setup ). This tool should

• take the Rust path as an argument
• have an option to choose header or source generation
• have an option to pick a custom C++ namespace prefix

This could look like

cxxqtbuilder --header --cpp-namespace-prefix=cxx_qt src/mod/object.rs > path/to/build/gen/object.h
cxxqtbuilder --cpp-namespace-prefix=cxx_qt src/mod/object.rs > path/to/build/gen/object.cpp

It also should have an option for generating cxx-qt-lib sources

cxxqtbuilder --generate-cxx-qt-lib-sources --out-dir path/to/build/lib/

CMake configure

This tool can then be wrapped by a CMake helper

cxx_qt_gen(
  GENERATED_SOURCES
  src/mod/object.rs
  src/mod/another.rs
  OUTPUT_DIRECTORY path/to/output/directory
)

add_executable(
  appbinary
  src/main.cpp
  src/qml.qrc
  ${GENERATED_SOURCES}
)

OUTPUT_DIRECTORY would be optional and default to (a subdirectory of) CMAKE_CURRENT_BINARY_DIR. There could be an optional NAMESPACE argument for the C++ namespace too.

CMake Build

For the building part we may be able to use https://github.com/corrosion-rs/corrosion to build and link the Rust crate to our Qt binary. As this would be a typical build now.

Problems / Further discussions

• Can we pass the C++ namespace prefix to cargo as an input env var from CMake? At the moment this is written to a special file and then read back. Or do we even need this long term, with extern Qt the namespace is specified in the bridge definition?
• Should the cxx-qt-lib generate sources be in their own sub folder and the project generation in another?
• If the Rust files change we need configure stage to happen again, can we force CMake to always run a configure step on every build?
• What happens with QQmlExtensionPlugin ?
• https://github.com/jason-yau/CxxRust may also be useful to look at / compare

[Be-ing]

cxxqtbuilder --header --cpp-namespace-prefix=cxx_qt src/mod/object.rs > path/to/build/gen/object.h
cxxqtbuilder --cpp-namespace-prefix=cxx_qt src/mod/object.rs > path/to/build/gen/object.cpp

An alternative CLI could be simpler:

cxxqtbuilder --cpp-namespace-prefix=cxx_qt -i src/mod/object.rs -o path/to/build/gen

This would generate both object.h and object.cpp in the path/to/build/gen directory instead of requiring two invocations of cxxqtbuilder for every file.

[Be-ing]

It turns out cxxbridge can be used with a single invocation: dtolnay/cxx#319 The interface is a bit different than I proposed, using separate -o arguments for each file instead of specifying a directory:

cxxbridge src/lib.rs -o lib.rs.h -o lib.rs.cc

[ahayzen-kdab]

Aha cool ! So yes cxxqtbridge src/mod/myobject.rs -o myobject.h -o myobject.cpp could work (note that we generate the .h and .cpp then the CXX bridge will run over the code again and generate the .rs.h and .rs.cc which are the actual bridge files)

[LeonMatthesKDAB]

Side note:
A good reason for being able to simply print the output of cxxqtbridge to stdout is that it makes it easy to test the tool itself.
Then you only need an expected file somewhere in your tests and don't have to manage temporary files that are created when running the tool.
So it's probably not necessary for end users, but might still be beneficial for ourselves.

And end users might use that to double-check the generated output as well, without messing with their current files.

[ahayzen-kdab]

Good point, so we could have cxxqtbridge --header src/lib.rs and cxxqtbridge --source src/lib.rs which output to stdout either the header or source? It also depends how much we want to test the cmd tool, as we have tests for the methods it uses.

[Be-ing]

Considering there are 4 different generated output files (cxx header and cpp, cxx-qt header and cpp), I think we should go with simply specifying an output directory after all. Requiring 4 separate paths to be passed to the CLI seems a bit silly.

[Be-ing]

cxx_qt_gen_out_dir(
  path/to/output/folder  # can we auto generate this? or if they are fast and need to be rerun on every build anyway can we just build into a hashed dir?
)
cxx_qt_gen_sources(
  GENERATED_SOURCES
  src/mod/object.rs
  src/mod/another.rs
)

How about combining these into one CMake function?

cxx_qt_gen(
  GENERATED_SOURCES
  src/mod/object.rs
  src/mod/another.rs
  OUTPUT_DIRECTORY path/to/output/directory
)

OUTPUT_DIRECTORY would be optional and default to (a subdirectory of) CMAKE_CURRENT_BINARY_DIR. There could be an optional NAMESPACE argument for the C++ namespace too.

[ahayzen-kdab]

If that makes sense in CMake, i'm not too familiar with how optional arguments work with CMake 😅 Looks good to me if that works though :-D and right a sub dir of current binary dir would work by default.

[LeonMatthes]

Should be possible with CMake and ist best-practice. So lets just go with @Be-ing 's suggestion

[LeonMatthesKDAB]

I've updated the main comment with your suggestion.

[LeonMatthesKDAB]

Hm, is there a way to "Resolve" this conversation?
That would be helpful, so we can see where there's still discussion 🤔

[ahayzen-kdab]

You can "hide" it, but not sure if it hides it for everyone...

[Be-ing]

Can we pass the C++ namespace prefix to cargo as an input env var from CMake? At the moment this is written to a special file and then read back. Or do we even need this long term, with extern Qt the namespace is specified in the bridge definition?

I think it would be ideal to have the namespace specified within the source code. If you want to change the namespace, it would be odd to have to change the build system to do that.

[ahayzen-kdab]

Right, I think we should remove it and go the same route as CXX when we do #147 so then it's an attribute either on the mod block or on the type/struct itself. This will simplify things here.

[LeonMatthesKDAB]

Then I guess we don't need the option in the build tool. Let's just stick with a default namespace name for now.

[ahayzen-kdab]

Right we are not going to add support for namespaces in the new tool, this will come later via macros.

[Be-ing]

Should the cxx-qt-lib generate sources be in their own sub folder and the project generation in another?

Could we turn cxx-qt-lib into a separate library installed along with the cxxqtbuilder tool? IIUC the generated code for cxx-qt-lib isn't changing per application. I'm not sure if this is feasible or desirable though.

[jason-yau]

I mean, can this part of work be done in CLI to solve the problem of duplicated symbol errors caused by multiple staticlibs? Can we use CLI to help us link those into a big staticlibs?

[Be-ing]

That is beyond the scope of the CLI tool discussed here which will just be generating C++ code. Cargo and rustc can already combine multiple rlibs into a staticlib. I haven't looked into the implementation details, but I think you just need a dummy crate for the staticlib with all the Rust libraries as dependencies in Cargo.toml.

[ahayzen-kdab]

Initially the CLI is just for generating the C++ code for the bridge, we would still use CMake to do the linking (and maybe look at corrosion etc for building + linking Rust). So although related the solution may be part of another discussion / change / improvement.

We also need to check that we can avoid multiple copies of CXX things colliding as well as our own.

[jason-yau]

@Be-ing That may be a good way to solve it. But is there a better way for this to be done automatically, rather than coding a dummy crate by developers?

[Be-ing]

I doubt it, but I will explore this more after we have the CLI tool basically working.

[ahayzen-kdab]

I also wonder if we should drop the generation stuff for QQmlExtensionPlugin and just require you to do it manually, as it's just a simple C++ file + qmldir file 🤔

[LeonMatthesKDAB]

Just a quick thought that came to mind to potentially make implementing this easier:

Technically, the only thing that we really need to do at configure time is figure out which files are relevant for the build.
Therefore, for an initial implementation, all that the CLI would need to do would be to touch all files, and return a list of them.
Then, when cmake --build . is run, we could run cargo build as a pre-build command.
This would then populate the files, causing all dependent files to be rebuilt as well.

I haven't thought this through entirely, especially with regards to the OUT_DIR stuff.
But might be an easy way to get started.

@Be-ing As it seems you'll be the one implementing this, also feel free to ignore my ramblings :) Just something that popped into my head a few days ago.

[ahayzen-kdab]

Hmm i think that would potentially be more complicated, as the idea is that the CLI tool would just use the existing API in cxx-qt-build, the user in the CMake should be able to specify where the files end up (to avoid collisions), eg cxxqtbridge src/lib.rs -o ${CMAKE_CURRENT_BINARY_DIR}/lib.h -o ${CMAKE_CURRENT_BINARY_DIR}/lib.cpp.

This then results in applications not needing the build.rs as the CLI tool does all the the C++ generation side, the Rust cargo build side just needs to do the macro expansion. If we just touched the files we could need to communicate to Rust where the files are located etc, which probably complicates things as it likely won't be the OUT_DIR but somewhere in the CMake build folder.

We just need to ensure that CXX is OK with the location and is able to pick up files for includes (hopefully just adding target_includes_directory should be enough).

[Be-ing]

Now that I have implemented the C++ code generation with the CLI tool in #161, I realize that building the C++ code that cxx generates from cxx-qt-lib as a library is a prerequisite for using the CLI tool in the build process. Currently, CxxQtBuilder invoked from build.rs is used to dump the C++ source code generated from cxx-qt-lib, which is embedded into the cxx-qt-build Rust library, to the Cargo target directory. This is quite hacky and inefficient. This process builds that generated C++ code repeatedly, once for each example/test. This would not scale well to a large application using many cxx-qt crates. I don't want to replicate this when integrating the CLI tool into the build process.

I hacked up a quick proof-of-concept CMakeLists.txt to build the C++ code generated from cxx-qt-lib and link it to Qt. This worked fine. I propose moving to this process for building this repo:

1. Build the CLI tool with Cargo
2. Configure CMake. CMake invokes Cargo at configure time to generate the C++ sources with cxx_gen::generate_header_and_cc from cxx-qt-lib's build.rs. The generated C++ sources are used to create a CMake library target.
3. Applications' CMakeLists.txt call find_package(CxxQt), which brings the library target and CMake helper functions into scope. Link the library to the application. After this is implemented, cxx-qt-build's dependency on cxx-qt-lib can be removed.
4. Applications' CMakeLists.txt call a macro that invokes the CLI tool to generate their C++ files at configure time
5. CMake invokes Cargo at build time to build the Rust crate(s), which are linked to the application

The C++ files generated from cxx-qt-lib would be packaged together with a CMakeLists.txt, the cxx.h header, a CMake config file for find_package, and the CMake module with the helper functions for downstream applications. Creating this package could be done by cxx-qt-lib's build.rs with a Cargo feature or with an xtask. This library could then be distributed together with the CLI tool. I could work on packaging that for vcpkg after this is implemented.

[ahayzen-kdab]

Hmm, so i'm not sure we can have CMake at configure time trigger a sub crate's build.rs like cxx-qt-lib's one and then find the output from it (this is one of the problems the cli tool is trying to fix). But I see your point of only generating the cxx-qt-lib output once.

So I was thinking it would work like this ...

• Install the CLI tool from cargo cargo install cxx-qt-cli or whatever it ends up as
• Applications have a CMakeLists.txt that has a find_package(CxxQt) in it (the CxxQt.cmake is either distributed manually or by them running maybe something like cxx-qt-cli --new-project /path/to/project)
  • CMake configure invokes the CLI tool to run something like cxxqtbridge --generate-statics --output /path/to/common/statics/folder (so generate the statics to a common output folder could be like build/statics which would then have like build/statics/cxx-qt-lib/qt_types.h
  • This is then made into a library target as you say, and is done once "globally" across the CMake project (probably triggered either by the find_package or via a call you do once somewhere in the root of your CMake like enable_cxx_qt() if we need to set the output path, maybe just having a variable you set for a custom folder like CXX_QT_STATIC_HEADERS_DIR before the find_package would be enough and it can default to CMAKE_BUILD_DIR/cxx-qt-statics)
• Applications then have a line which calls the CMake helper to run the cxxqtbridge for each of their Rust files, outputting the C++ into a folder of their choice (eg CMAKE_CURRENT_BUILD_DIRECTORY or whatever)
• CMake build invokes cargo build, which builds the staticlib and then is linked with the C++ app etc

This then means that developers will only need to install cxx-qt-cli from cargo, everything else will be managed for them. The cxx-qt-cli tool could even be called from CxxQt.cmake to ensure that the CxxQt.cmake is up to date and request the developer to use the CLI tool to update the cmake file.

[Be-ing]

Sorry the way I originally wrote that was a bit ambiguous. I meant generate the cxx-qt-lib C++ files at configure time for this repo's CMakeLists.txt, not for downstream applications.

[ahayzen-kdab]

Ah :-) So the idea I thought would be the next pull requests would be moving the examples/tests to use the CLI tool in the same way downstream applications would use it? (eg make something do cargo install ./cxx-qt-cli first to get the latest CLI binary, then run the rest of CMake and have them generating files into build folders rather than target etc)

Then once that is done all the Rust builds should be able to share the same cargo workspace so we'd only build cxx-qt-lib once ?

But maybe I'm missing the advantage / what you are expressing as a proposed change :-)

[Be-ing]

Roughly yes, the top CMakeLists.txt in this repo would generate the C++ files from cxx-qt-lib and add them to a CMake library target shared by all the tests and examples.

I'm still not fond of putting whole C++ source code files into static strs in the executable as a means of distributing a C++ library. It would be awkward to avoid recompiling the C++ code multiple times this way. I suppose it would be possible by checking if the CMake library target already exists in the CxxQt.cmake module before creating it, but that feels hacky. Plus, that logic would have to be duplicated if someone wants to use a different build system with cxx-qt. Additionally, that would still require downstream applications to vendor the CxxQt.cmake module. So I think it would be best to distribute the cxx-qt-lib C++ code as a C++ library like any other.

[Be-ing]

Follow up on this for posterity: I successfully implemented this in #173, then realized it wasn't a great idea after all due to the practicalities of distributing a Rust executable, a C++ library, and a CMake module which all have to have exactly the same version. Instead, in #174 and #218 I refactored cxx-qt-build so that it not only generates C++ code, but also compiles it, links it with the Rust, and links Qt. This greatly simplified the CMake integration; we don't need our own custom CMake module anymore and instead only need a few calls to Corrosion. This also allows much easier integration with other C++ build systems. All the C++ build system has to do is call cargo build with the CXXQT_EXPORT_DIR and QMAKE environment variables set, add the CXXQT_EXPORT_DIR to the C++ include paths, and link the Rust library. As a bonus, this also allows doing the entire build of an executable with Cargo without needing CMake at all (#221).