This document describes the technical architecture of Rerun.
It all starts with logging. You can log rich data (point clouds, images, etc) with either our Python SDK or our Rust SDK.
The logging SDK:s encodes the data using Apache Arrow (see more below).
The logging data can be written to disk as .rrd files, or transmitted over TCP to either a Rerun Viewer or a Rerun Server.
The Rerun Viewer is where log data is visualized. It is usually run as a native app, but can also be compiled to WebAssembly (Wasm) and run in a browser.
The easiest way to launch the viewer is directly from the logging API with rr.init("my_app", spawn=True). However, the standalone viewer can also be run from the command line, for example to view an .rrd file: rerun mydata.rrd.
You can try running the viewer in a browser using rr.serve() in python, or using rerun --web-viewer mydata.rrd.
The web viewer consists of just a few small files - a thin .html, a .wasm blob, and an auto-generated .js bridge for the wasm. These files are served using the re_web_viewer_server crate.
The web viewer can load .rrd files (just drag-drop them into the browser), or read logging data streamed over WebSockets.
.rrd ("Rerun Data") is just a bunch of log messages appended one after the other to a file.
NOTE: .rrd files do not yet guarantee any backwards or forwards compatibility. One version of Rerun will likely not be able to open an .rrd file generated by another Rerun version.
Apache Arrow is a language-independent columnar memory format for arbitrary data. We use it to encode the log data when transmitting it over the network or storing it in an .rrd file. We also use it in our in-RAM data store, re_arrow_store.
In rust, we use the arrow2 crate.
The Rerun Viewer uses the wgpu graphics API. It provides a high-performance abstraction over Vulkan, Metal, D3D12, D3D11, OpenGLES, WebGL and WebGPU. This lets us write the same code graphics code for native as for web.
We use the WebGL backend when compiling for web. Once WebGPU is available in most browsers, we can easily switch to it for a nice performance boost!
We have written our own high-level rendering crate on top of wgpu, called re_renderer.
The GUI in the Rerun Viewer is using egui, a cross-platform, immediate mode GUI.
We use eframe, the egui framework, to run egui on both native and web.
Wasm (short for WebAssembly) is a binary instruction format supported by all major browser. The Rerun Viewer can be compiled to Wasm and run in a browser.
Threading support in Wasm is nascent, so care must we taken that we don't spawn any threads when compiling for wasm32.
Wasm has no access to the host system, except via JS calls (something that may change once WASI rolls out), so when compiling for wasm32 you can NOT use the Rust standard library to:
- Access files
- Read environment variables
- Get the current time (use
instantinstead) - Use networking (use
ehttp,reqwest, orewebsockinstead) - etc
The Rerun Viewer uses an immediate mode GUI, egui. This means that each frame the entire GUI is being laid out from scratch.
In fact, the whole of the Rerun Viewer is written in an immediate mode style. Each rendered frame it will query the in-RAM data store, massage the results, and feed it to the renderer.
The advantage of immediate mode is that is removes all state management. There is no callbacks that are called when some state has already changed, and the state of the blueprint is always in sync with what you see on screen.
Immediate mode is also a forcing function, forcing us to relentlessly optimize our code. This leads to a very responsive GUI, where there is no "hickups" when switching data source or doing time scrubbing.
Of course, this will only take us so far. In the future we plan on caching queries and work submitted to the renderer so that we don't perform unnecessary work each frame. We also plan on doing larger operation in background threads. This will be necessary in order to support viewing large datasets, e.g. several million points. The plan is still to do so within an immediate mode framework, retaining most of the advantages of stateless code.