otel: Add OpenTelemetry functionality to NGINX Unit #1463

javorszky · 2024-10-18T12:43:04Z

Adds OpenTelemetry implementation via a Rust crate compiled to a static C library and linked into the existing Unit codebase with the necessary build and configuration steps.

Each commit in order builds, runs, and works as intended, each export traces to grafana from 290fffd (configuration items and their validation) onwards
The --otel build argument works from fb15df7 (add build tooling to include otel code) onwards.
Adding settings.telemetry to Unit's config.json works from 290fffd (configuration items and their validation) onwards
Each commit has the Signed-off-by trailers, and all of them are co-authored between @avahahn and myself.

Closes #1283

avahahn · 2024-10-21T15:17:36Z

Opentelemetry performance test results!

The tests use wrk, were ran on a 12c/24t system with 64gb ram. wrk invocation as follows:
./wrk -t22 -c880 -d30s http://127.0.0.1:80/
Here, Unit is serving the welcome page.

With otel change, tracing enabled (sample rate 1.0, batch size 20, HTTP Transport)

Latency:

Avg: 1.55ms
Stdev: 2.67ms, 95.46%

Requests per second:

Avg: 31.70k
Stdev: 5.87k, 79.66%

With otel change, tracing NOT enabled:

Latency:

Avg: 1.00ms
Stdev: 777.66us 75.06%

Requests per second:

Avg: 41.76k
Stdev: 11.33k 55.51%

Built without OpenTelemetry support:

Latency:

Avg: 0.97ms
Stdev: 673.74us, 77.44%

Requests per second:

Avg: 42.25k
Stdev: 4.88k, 76.60%

I conclude that the enclosed change does not measurably impact Unit's performance when not configured. The differences in numbers between the 2nd and 3rd dataset is far smaller than either of their deviance values, and I find the data is always similar across test runs.

I also conclude that in a very high throughput deployment opentelemetry tracing will have a small impact. This test represents a high throughput deployment with the sample ratio cranked up unreasonably high. Best practices in any kind of situation in which unit will be receiving this kind of traffic are to reduce the sampling ratio to minimize the load of opentelemetry span processing and generation. Even with the sampling ratio cranked up to maximum (100%) opentelemetry processing for every single request (over 30k requests per second) merely adds half a millisecond on average.

For more information you can reproduce my tests with these scripts:
https://github.com/avahahn/unit-perf-test

avahahn · 2024-10-21T19:09:46Z

Just wanted to offer a note for configuration. The settings object has been extended with a new telemetry field. This field contains currently 4 items:

endpoint: The endpoint for the OpenTelemetry Collector
This is a required field.
It takes either a URL to either a gRPC or HTTP API.
Example: http://lgtm:4318/v1/traces
protocol: Determines the protocol used to communicate with the endpoint
This is a required field.
Can either be specified in all caps or all lowercase.
One of either "HTTP" or "GRPC"
batch_size: Number of spans to cache before triggering a transaction with configured endpoint
This allows the user to cache up to N spans before the opentelemetry background thread sends spans over network to the collector. It is expected that this scales up with the throughput of the user's environment as well as given any bandwidth constraints the user might have for their network.
Must be a positive integer.
sampling_ratio: percent of requests to trace
This allows the user to only trace anywhere from 0% to 100% of requests that hit Unit. In high throughput environments this percentage will be lower. This allows the user to save space in storing span data, and to collect request metrics like time to decode headers and whatnot without storing massive amounts of duplicate superfluous data.
Must be a positive floating point number.

Example configuration:

curl -X PUT 127.0.0.1:8080/config -d '{
    "settings": {
        "telemetry": {
            "batch_size": 20,
            "endpoint": "http://lgtm:4318/v1/traces",
            "protocol": "http",
            "sampling_ratio": 1
        }
    },
    "listeners": {
        "*:80": {
            "pass": "routes"
        }
    },
    "routes": [
        {
            "match": {
                "headers": {
                    "accept": "*text/html*"
                }
            },
            "action": {
                "share": "/usr/share/unit/welcome/welcome.html"
            }
        },
        {
            "action": {
                "share": "/usr/share/unit/welcome/welcome.md"
            }
        }
    ]
}'

ac000

For the commits with Ava's Co-authored-by, the commit tags should look like

Co-developed-by: Ava Hahn <a.hahn@f5.com>
Signed-off-by: Ava Hahn <a.hahn@f5.com>
Signed-off-by: Gabor Javorszky <g.javorszky@f5.com>

I.e. A c-[ad]-b tag should be immediately followed by a s-o-b from the same person.

Also note we use Co-developed-by.

For the

tools: fix bracket balance of editorconfig file

commit. tools is really for the tools/ directory. Here i'd just use the filename as the prefix, i.e.

.editorconfig: fix bracket imbalance

ac000

In commit

    otel: add build tooling to include otel code

I think the following would be better in ./configure

diff --git ./auto/make ./auto/make
index f21a2dfc..e7d29ba3 100644
--- ./auto/make
+++ ./auto/make
@@ -7,6 +7,11 @@
 
 $echo "creating $NXT_MAKEFILE"
 
+if [ $NXT_OTEL = "NO" ];  then
+    NXT_OTEL_LIB_LOC=
+    NXT_OTEL_BUILD_FLAG=
+    NXT_OTEL_LIB_DIR=
+fi
 
 cat << END > $NXT_MAKEFILE
 
@@ -138,14 +143,14 @@ cat << END >> $NXT_MAKEFILE
 
 libnxt:        $NXT_BUILD_DIR/lib/$NXT_LIB_SHARED $NXT_BUILD_DIR/lib/$NXT_LIB_STATIC
 
-$NXT_BUILD_DIR/lib/$NXT_LIB_SHARED: \$(NXT_LIB_OBJS)
+$NXT_BUILD_DIR/lib/$NXT_LIB_SHARED: \$(NXT_LIB_OBJS) $NXT_OTEL_LIB_LOC
        \$(PP_LD) \$@
        \$(v)\$(NXT_SHARED_LOCAL_LINK) -o \$@ \$(NXT_LIB_OBJS) \\
-               $NXT_LIBM $NXT_LIBS $NXT_LIB_AUX_LIBS
+               $NXT_LIBM $NXT_LIBS $NXT_LIB_AUX_LIBS $NXT_OTEL_LIB_LOC
 
-$NXT_BUILD_DIR/lib/$NXT_LIB_STATIC: \$(NXT_LIB_OBJS)
+$NXT_BUILD_DIR/lib/$NXT_LIB_STATIC: \$(NXT_LIB_OBJS) $NXT_OTEL_LIB_LOC
        \$(PP_AR) \$@
-       \$(v)$NXT_STATIC_LINK \$@ \$(NXT_LIB_OBJS)
+       \$(v)$NXT_STATIC_LINK \$@ \$(NXT_LIB_OBJS) $NXT_OTEL_LIB_LOC
 
 $NXT_BUILD_DIR/lib/$NXT_LIB_UNIT_STATIC: \$(NXT_LIB_UNIT_OBJS) \\
                $NXT_BUILD_DIR/share/pkgconfig/unit.pc \\
@@ -359,11 +364,11 @@ $echo >> $NXT_MAKEFILE
 cat << END >> $NXT_MAKEFILE
 
 $NXT_BUILD_DIR/sbin/$NXT_DAEMON:       $NXT_BUILD_DIR/lib/$NXT_LIB_STATIC \\
-                               \$(NXT_OBJS)
+                               \$(NXT_OBJS) $NXT_OTEL_LIB_LOC
        \$(PP_LD) \$@
        \$(v)\$(NXT_EXEC_LINK) -o \$@ \$(CFLAGS) \\
                \$(NXT_OBJS) $NXT_BUILD_DIR/lib/$NXT_LIB_STATIC \\
-               $NXT_LIBM $NXT_LIBS $NXT_LIB_AUX_LIBS
+               $NXT_LIBM $NXT_LIBS $NXT_LIB_AUX_LIBS $NXT_OTEL_LIB_LOC
 
 END

Where you only need to adjust the NXT_LIB_AUX_CFLAGS and NXT_LIB_AUX_LIBS variables.

E.g. in my http compression patches I do

diff --git ./configure ./configure
index 6929d41d..f33134b7 100755
--- ./configure
+++ ./configure
@@ -127,6 +127,7 @@ NXT_LIBRT=
 . auto/unix
 . auto/os/conf
 . auto/ssltls
+. auto/compression
 
 if [ $NXT_REGEX = YES ]; then
     . auto/pcre
@@ -169,11 +170,13 @@ END
 
 NXT_LIB_AUX_CFLAGS="$NXT_OPENSSL_CFLAGS $NXT_GNUTLS_CFLAGS \\
                     $NXT_CYASSL_CFLAGS $NXT_POLARSSL_CFLAGS \\
-                    $NXT_PCRE_CFLAGS"
+                    $NXT_PCRE_CFLAGS $NXT_ZLIB_CFLAGS $NXT_ZSTD_CFLAGS \\
+                    $NXT_BROTLI_CFLAGS"
 
 NXT_LIB_AUX_LIBS="$NXT_OPENSSL_LIBS $NXT_GNUTLS_LIBS \\
                     $NXT_CYASSL_LIBS $NXT_POLARSSL_LIBS \\
-                    $NXT_PCRE_LIB"
+                    $NXT_PCRE_LIB $NXT_ZLIB_LIBS $NXT_ZSTD_LIBS \\
+                    $NXT_BROTLI_LIBS"
 
 if [ $NXT_NJS != NO ]; then
     . auto/njs

auto/make

auto/otel

src/nxt_otel.c

auto/otel

src/nxt_otel.c

ac000 · 2024-10-22T16:04:46Z

For

tools/openapi: update OpenAPI references

I would make the prefix docs/openapi as docs/unit-openapi.yaml is the source of truth.

ac000

otel: configuration items and their validation

Adds code responsible for users to apply the `telemetry` configuration
options.

I would like to see example Unit configuration here...

ac000 · 2024-10-22T16:07:54Z

src/nxt_conf_validation.c

+#if (NXT_HAVE_OTEL)
+nxt_inline nxt_int_t nxt_otel_validate_endpoint(nxt_conf_validation_t *vldt,
+                                                nxt_conf_value_t *value,
+                                                void *data);
+nxt_int_t nxt_otel_validate_batch_size(nxt_conf_validation_t *vldt,
+                                       nxt_conf_value_t *value,
+                                       void *data);
+nxt_int_t nxt_otel_validate_sample_ratio(nxt_conf_validation_t *vldt,
+                                         nxt_conf_value_t *value,
+                                         void *data);
+nxt_int_t nxt_otel_validate_protocol(nxt_conf_validation_t *vldt,
+                                     nxt_conf_value_t *value,
+                                     void *data);
+#endif


These should be static

ac000 · 2024-10-22T16:09:01Z

src/nxt_conf_validation.c

@@ -1465,6 +1515,73 @@ nxt_conf_validate(nxt_conf_validation_t *vldt)
    "a number, a string, an array, or an object"


+
+#if (NXT_HAVE_OTEL)
+inline nxt_int_t


Remove the inline, or simply remove this function altogether is probably better...

ac000 · 2024-10-22T16:09:48Z

src/nxt_conf_validation.c

+    double batch_size;
+    batch_size = nxt_conf_get_number(value);
+    if (batch_size <= 0) {
+      return NXT_ERROR;


Oops, indentation...

ac000 · 2024-10-22T16:10:14Z

src/nxt_conf_validation.c

+    double batch_size;
+    batch_size = nxt_conf_get_number(value);


Blank line after variable declarations please.

ac000 · 2024-10-22T16:10:55Z

src/nxt_conf_validation.c

+
+    return NXT_ERROR;
+
+ happy:


Don't indent goto labels...

ac000 · 2024-10-22T16:12:50Z

src/nxt_conf_validation.c

+    if (nxt_str_eq(&proto, "HTTP", 4) ||
+        nxt_str_eq(&proto, "http", 4)) {


In Unit we put the operator on the next line and when we have multi-line if's the { goes on it its own line.

ac000 · 2024-10-22T16:14:52Z

src/nxt_router.c

@@ -1613,6 +1617,12 @@ static nxt_conf_map_t  nxt_router_websocket_conf[] = {
 };


+#if (NXT_HAVE_OTEL)
+static void nxt_otel_log_callback(u_char *arg) {


Opening { of functions always on their own line please...

ac000 · 2024-10-22T16:25:05Z

src/nxt_router.c

+        nxt_conf_get_string(otel_endpoint, &telemetry_endpoint);
+        nxt_conf_get_string(otel_proto, &telemetry_proto);
+        telemetry_batching = otel_batching ? nxt_conf_get_number(otel_batching) : NXT_OTEL_BATCH_DEFAULT;
+        telemetry_sample_fraction = otel_sampling ? nxt_conf_get_number(otel_sampling) : NXT_OTEL_SAMPLING_DEFAULT;


Need to break this line up...

ac000 · 2024-10-22T16:25:57Z

src/nxt_router.c

+        nxt_otel_rs_init(&nxt_otel_log_callback,
+                      &telemetry_endpoint,
+                      &telemetry_proto,
+                      telemetry_sample_fraction,
+                      telemetry_batching);


Could do with a wee re-alignment to the opening (

I wonder also if these are just asking to be put in a struct?

src/nxt_router.c

src/nxt_otel.c

This is purely the source code of the rust end of opentelemetry. It does not have build tooling wired up yet, nor is this used from the C code. Signed-off-by: Ava Hahn <a.hahn@f5.com> Signed-off-by: Gabor Javorszky <g.javorszky@f5.com> Co-authored-by: Ava Hahn <a.hahn@f5.com>

src/nxt_otel.c

+    body_key = (nxt_str_t){
+        .start  = body_size_buf,
+        .length = nxt_length(body_size_buf),
+    };
+    body_val = (nxt_str_t){
+        .start  = body_buf,
+        .length = nxt_length(body_buf),
+    };


src/nxt_otel.c

src/nxt_otel.h

Adds the --otel flag to the configure command and the various build time variables and checks that are needed in this flow. It also includes the nxt_otel.c and nxt_otel.h files that are needed for the rest of Unit to talk to the compiled static library that's generated from the rust crate. Signed-off-by: Ava Hahn <a.hahn@f5.com> Signed-off-by: Gabor Javorszky <g.javorszky@f5.com> Co-authored-by: Ava Hahn <a.hahn@f5.com>

Enables Unit to parse the tracestate and traceparent headers and add it to the list, as well as calls to nxt_otel_test_and_call_state. Signed-off-by: Ava Hahn <a.hahn@f5.com>

Adds code responsible for users to apply the `telemetry` configuration options. configuration snippet as follows: { "settings": { "telemetry": { "batch_size": 20, "endpoint": "http://lgtm:4318/v1/traces", "protocol": "http", "sampling_ratio": 1 } }, "listeners": { "*:80": { "pass": "routes" } }, "routes": [ { "match": { "headers": { "accept": "*text/html*" } }, "action": { "share": "/usr/share/unit/welcome/welcome.html" } }, { "action": { "share": "/usr/share/unit/welcome/welcome.md" } } ] } Signed-off-by: Ava Hahn <a.hahn@f5.com> Signed-off-by: Gabor Javorszky <g.javorszky@f5.com> Co-authored-by: Ava Hahn <a.hahn@f5.com>

Tiny bracket balance fix. Signed-off-by: Ava Hahn <a.hahn@f5.com> Signed-off-by: Gabor Javorszky <g.javorszky@f5.com>

These changes are generated by the openapi generator through a make command. Signed-off-by: Ava Hahn <a.hahn@f5.com> Signed-off-by: Gabor Javorszky <g.javorszky@f5.com>

ac000 · 2024-10-25T02:26:18Z

src/otel/src/lib.rs

+
+#[repr(C)]
+pub struct nxt_str_t {
+    pub len: usize,


Could we make this length please?, so it matches the C code (and wasm-wasi-component).

javorszky requested review from avahahn and ac000 October 18, 2024 12:43

javorszky force-pushed the gabor/otel-review-2-check branch from bb5da68 to dac737c Compare October 21, 2024 12:12

ac000 requested changes Oct 21, 2024

View reviewed changes

ac000 requested changes Oct 22, 2024

View reviewed changes

ac000 reviewed Oct 22, 2024

View reviewed changes

src/nxt_otel.c Outdated Show resolved Hide resolved

avahahn force-pushed the gabor/otel-review-2-check branch from 51d0c1b to dac737c Compare October 22, 2024 22:34

avahahn force-pushed the gabor/otel-review-2-check branch from dac737c to 6ebfe70 Compare October 22, 2024 22:35

ac000 requested changes Oct 22, 2024

View reviewed changes

ac000 reviewed Oct 23, 2024

View reviewed changes

src/nxt_otel.h Outdated Show resolved Hide resolved

ac000 mentioned this pull request Oct 23, 2024

Use nxt_nitems() instead of sizeof() for strings (arrays) #1467

Open

avahahn force-pushed the gabor/otel-review-2-check branch from 6ebfe70 to c2d6ce7 Compare October 23, 2024 21:19

This comment was marked as duplicate.

Sign in to view

javorszky and others added 5 commits October 23, 2024 16:33

otel: add header parsing and test call state

2ef8f21

Enables Unit to parse the tracestate and traceparent headers and add it to the list, as well as calls to nxt_otel_test_and_call_state. Signed-off-by: Ava Hahn <a.hahn@f5.com>

.editorconfig: fix bracket balance of editorconfig file

efffea4

Tiny bracket balance fix. Signed-off-by: Ava Hahn <a.hahn@f5.com> Signed-off-by: Gabor Javorszky <g.javorszky@f5.com>

docs/openapi: update OpenAPI references

ee3b440

These changes are generated by the openapi generator through a make command. Signed-off-by: Ava Hahn <a.hahn@f5.com> Signed-off-by: Gabor Javorszky <g.javorszky@f5.com>

avahahn force-pushed the gabor/otel-review-2-check branch from c2d6ce7 to ee3b440 Compare October 23, 2024 23:33

ac000 reviewed Oct 25, 2024

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

otel: Add OpenTelemetry functionality to NGINX Unit #1463

otel: Add OpenTelemetry functionality to NGINX Unit #1463

javorszky commented Oct 18, 2024 •

edited

Loading

avahahn commented Oct 21, 2024

avahahn commented Oct 21, 2024

ac000 left a comment

ac000 left a comment

ac000 commented Oct 22, 2024

ac000 left a comment

ac000 Oct 22, 2024

ac000 Oct 22, 2024

ac000 Oct 22, 2024

ac000 Oct 22, 2024

ac000 Oct 22, 2024

ac000 Oct 22, 2024

ac000 Oct 22, 2024

ac000 Oct 22, 2024

ac000 Oct 22, 2024

ac000 Oct 23, 2024

This comment was marked as duplicate.

This comment was marked as duplicate.

ac000 Oct 25, 2024

		if (nxt_str_eq(&proto, "HTTP", 4) \|\|
		nxt_str_eq(&proto, "http", 4)) {

otel: Add OpenTelemetry functionality to NGINX Unit #1463

Are you sure you want to change the base?

otel: Add OpenTelemetry functionality to NGINX Unit #1463

Conversation

javorszky commented Oct 18, 2024 • edited Loading

avahahn commented Oct 21, 2024

Opentelemetry performance test results!

With otel change, tracing enabled (sample rate 1.0, batch size 20, HTTP Transport)

With otel change, tracing NOT enabled:

Built without OpenTelemetry support:

avahahn commented Oct 21, 2024

ac000 left a comment

Choose a reason for hiding this comment

ac000 left a comment

Choose a reason for hiding this comment

ac000 commented Oct 22, 2024

ac000 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

This comment was marked as duplicate.

This comment was marked as duplicate.

Choose a reason for hiding this comment

javorszky commented Oct 18, 2024 •

edited

Loading