Skip to content

Replace Logging JWT perf test with Locust - #8201

Merged
Amaury Chamayou (achamayou) merged 9 commits into
mainfrom
achamayou-logging-jwt-locust-benchmark
Aug 25, 2026
Merged

Replace Logging JWT perf test with Locust#8201
Amaury Chamayou (achamayou) merged 9 commits into
mainfrom
achamayou-logging-jwt-locust-benchmark

Conversation

@achamayou

@achamayou Amaury Chamayou (achamayou) commented Aug 24, 2026

Copy link
Copy Markdown
Member

Summary

  • replace the Piccolo-based pi_ls_jwt target with pi_logging_jwt_blocking_locust
  • benchmark JWT-authenticated POST /app/log/blocking/private calls that return only after commit
  • measure the logging workload at a 20ms signature interval and publish it under the unqualified Logging JWT Blocking Locust Bencher label
  • pass the temporary issuer's JWT through the Locust subprocess environment so it is not exposed in command lines or logs
  • extract the existing distributed-worker orchestration, steady-state timing, signature interval handling, failure checks, and Bencher reporting into shared Locust benchmark helpers
  • retain certificate-authenticated behavior for the existing Basic Blocking Locust benchmark

The Bencher label is now Logging JWT Blocking Locust rather than reusing Logging JWT, because the blocking-on-commit workload is not directly comparable to the previous pipelined Piccolo workload.

Validation

  • generated CTest command contains only --sig-ms-intervals 20
  • Locust workload discovery for both the logging JWT and basic blocking locustfiles
  • CLI rejection of a non-positive spawn rate
  • Black, Ruff, and gersemi checks

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
@achamayou
Amaury Chamayou (achamayou) requested a review from a team as a code owner August 24, 2026 14:17
Copilot AI lite review requested due to automatic review settings August 24, 2026 14:17

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR replaces the existing Piccolo-based logging JWT performance benchmark with a Locust-driven benchmark, and extracts shared Locust orchestration/stat parsing into reusable infra helpers so multiple blocking-on-commit workloads can share the same runner/reporting logic.

Changes:

  • Introduces shared Locust benchmark orchestration (tests/infra/locust_benchmark.py) plus Locust-side steady-state helpers (tests/infra/locust_benchmark_support.py) and unit tests.
  • Adds a new JWT-authenticated blocking logging Locust benchmark (tests/logging_jwt_locust.py + tests/infra/logging_jwt_locustfile.py) that passes the JWT via environment variable.
  • Refactors the existing Basic Blocking Locust benchmark to use the shared orchestration/helpers, and updates CMake targets accordingly.

Custom instructions used:

  • None from .github/copilot-instructions.md or .github/instructions/ were loaded during this review.

Reviewed changes

Copilot reviewed 8 out of 8 changed files in this pull request and generated 2 comments.

Show a summary per file
File Description
tests/logging_jwt_locust.py Adds the new top-level runner for the JWT blocking logging Locust benchmark.
tests/infra/logging_jwt_locustfile.py Defines the Locust workload for JWT-authenticated blocking logging writes.
tests/infra/locust_benchmark.py New shared Locust runner: CLI args, process orchestration, CSV parsing, Bencher reporting.
tests/infra/locust_benchmark_support.py New shared locustfile utilities: steady-state timing and TLS context creation.
tests/basicperf_locust.py Refactors Basic Blocking Locust benchmark runner to use the shared orchestration.
tests/infra/basicperf_locustfile.py Refactors the locustfile to reuse shared steady-state + TLS helpers.
tests/infra/test_locust_benchmark.py Adds unit tests for env-secret handling and stats parsing/result validation.
CMakeLists.txt Registers the new unit test and replaces the old Piccolo perf target with the new Locust perf target.

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Comment thread tests/infra/logging_jwt_locustfile.py
Comment thread tests/infra/locust_benchmark.py
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
@github-actions

github-actions Bot commented Aug 24, 2026

Copy link
Copy Markdown
Description

Comparing 5 available runs from this branch (#8201) against the trend of the last 30 main runs.

Each chart plots every benchmark as an axis, with values normalized so 100 is the EWMA baseline of recent main runs, using a 7-run half-life. The 5 orange branch lines run from the oldest (faintest) to the latest (darkest and thickest); the darker blue band is the main baseline +/- 1 std dev and the lighter blue band around it is +/- 2 std dev.

Axis labels show the latest branch value and its difference from the main EWMA baseline, where 0% is on the baseline. They are coloured green where the latest run improves on the baseline, red where it regresses, and grey where the difference is within one std dev of the baseline (within noise). Higher is better for throughput and rate, lower for latency and memory.

A benchmark which does not exist on main yet has no baseline of its own, so its earliest available run from this branch is used as its reference and its band is measured across this branch's runs. Its axis is normalized, scaled and coloured like any other, but the comparison is against this branch rather than against main.

Throughput (tx/s)

---
config:
  radar:
    width: 620
    height: 620
    marginTop: 90
    marginRight: 220
    marginBottom: 60
    marginLeft: 220
    axisLabelFactor: 1.12
    curveTension: 0.08
  theme: base
  themeCSS: |
    .radarCurve-0{fill:color-mix(in srgb, #62B5E5 13%, var(--color-canvas-default,var(--bgColor-default,#fff)))!important;fill-opacity:1!important;stroke:none!important;stroke-width:0!important}
    .radarCurve-1{fill:color-mix(in srgb, #62B5E5 40%, var(--color-canvas-default,var(--bgColor-default,#fff)))!important;fill-opacity:1!important;stroke:none!important;stroke-width:0!important}
    .radarCurve-2{fill:color-mix(in srgb, #62B5E5 13%, var(--color-canvas-default,var(--bgColor-default,#fff)))!important;fill-opacity:1!important;stroke:none!important;stroke-width:0!important}
    .radarCurve-3{fill:var(--color-canvas-default,var(--bgColor-default,#fff))!important;fill-opacity:1!important;stroke:none!important;stroke-width:0!important}
    .radarAxisLabel,.radarTitle{fill:var(--color-fg-default,var(--fgColor-default,#111827))!important;color:var(--color-fg-default,var(--fgColor-default,#111827))!important}
    .radarCurve-4{stroke-width:1.5px!important;stroke-opacity:0.40!important}
    .radarCurve-5{stroke-width:1.5px!important;stroke-opacity:0.50!important}
    .radarCurve-6{stroke-width:1.75px!important;stroke-opacity:1.00!important}
    .radarAxisLabel:nth-of-type(1){fill:#808A94!important}
    .radarAxisLabel:nth-of-type(2){fill:#808A94!important}
    .radarAxisLabel:nth-of-type(3){fill:#808A94!important}
    .radarAxisLabel:nth-of-type(4){fill:#808A94!important}
    .radarAxisLabel:nth-of-type(5){fill:#808A94!important}
    .radarAxisLabel:nth-of-type(6){fill:#808A94!important}
    .radarAxisLabel:nth-of-type(7){fill:#808A94!important}
    .radarAxisLabel:nth-of-type(8){fill:#808A94!important}
    .radarAxisLabel:nth-of-type(9){fill:#E5484D!important}
  themeVariables:
    cScale0: "#62B5E5"
    cScale1: "#62B5E5"
    cScale2: "#62B5E5"
    cScale3: "#62B5E5"
    cScale4: "#F97316"
    cScale5: "#F97316"
    cScale6: "#F97316"
    radar:
      axisColor: "#9CA3AF"
      graticuleColor: "#E5E7EB"
      graticuleOpacity: 0
      axisStrokeWidth: 1
      curveOpacity: 0
---
radar-beta
  axis b0["Basic: 69,715 tx/s ▬ -1%"]
  axis b2["Basic Blocking Locust 100ms: 3,050 tx/s ▬ 0%"]
  axis b3["Basic Blocking Locust 20ms: 15,147 tx/s ▬ 0%"]
  axis b4["Basic Blocking Locust 2ms: 31,265 tx/s ▬ -2%"]
  axis b5["Basic JS: 4,671 tx/s ▬ -1%"]
  axis b6["Basic Multi-Threaded: 84,903 tx/s ▬ -2%"]
  axis b7["Historical Queries: 205,228 tx/s ▬ -1%"]
  axis b8["Logging: 65,944 tx/s ▬ 0%"]
  axis b10["L…g JWT Blocking Locust: 14,789 tx/s ▼ 1%"]
  curve stddev2_high["main EWMA + 2 std dev"]{107.45, 100.57, 100.64, 108.77, 107.03, 106.82, 111.89, 108.50, 101.52}
  curve stddev1_high["main EWMA + 1 std dev"]{103.73, 100.29, 100.32, 104.39, 103.51, 103.41, 105.94, 104.25, 100.76}
  curve stddev1_low["main EWMA - 1 std dev"]{96.27, 99.71, 99.68, 95.61, 96.49, 96.59, 94.06, 95.75, 99.24}
  curve stddev2_low["main EWMA - 2 std dev"]{92.55, 99.43, 99.36, 91.23, 92.97, 93.18, 88.11, 91.50, 98.48}
  curve branch_2["#8201 (2 runs earlier)"]{103.81, 100.02, 100.01, 104.32, 100.86, 102.18, 98.97, 104.33, 100.00}
  curve branch_3["#8201 (1 run earlier)"]{101.99, 99.94, 99.95, 100.47, 101.44, 99.63, 104.40, 99.23, 98.16}
  curve branch_4["#8201"]{99.12, 99.88, 100.02, 97.56, 99.38, 98.13, 98.62, 99.97, 98.80}
  graticule polygon
  max 121
  min 79
  ticks 0
  showLegend false
Loading

Latency (ms)

---
config:
  radar:
    width: 620
    height: 620
    marginTop: 90
    marginRight: 220
    marginBottom: 60
    marginLeft: 220
    axisLabelFactor: 1.12
    curveTension: 0.08
  theme: base
  themeCSS: |
    .radarCurve-0{fill:color-mix(in srgb, #62B5E5 13%, var(--color-canvas-default,var(--bgColor-default,#fff)))!important;fill-opacity:1!important;stroke:none!important;stroke-width:0!important}
    .radarCurve-1{fill:color-mix(in srgb, #62B5E5 40%, var(--color-canvas-default,var(--bgColor-default,#fff)))!important;fill-opacity:1!important;stroke:none!important;stroke-width:0!important}
    .radarCurve-2{fill:color-mix(in srgb, #62B5E5 13%, var(--color-canvas-default,var(--bgColor-default,#fff)))!important;fill-opacity:1!important;stroke:none!important;stroke-width:0!important}
    .radarCurve-3{fill:var(--color-canvas-default,var(--bgColor-default,#fff))!important;fill-opacity:1!important;stroke:none!important;stroke-width:0!important}
    .radarAxisLabel,.radarTitle{fill:var(--color-fg-default,var(--fgColor-default,#111827))!important;color:var(--color-fg-default,var(--fgColor-default,#111827))!important}
    .radarCurve-4{stroke-width:1.5px!important;stroke-opacity:0.40!important}
    .radarCurve-5{stroke-width:1.5px!important;stroke-opacity:0.50!important}
    .radarCurve-6{stroke-width:1.75px!important;stroke-opacity:1.00!important}
    .radarAxisLabel:nth-of-type(1){fill:#808A94!important}
    .radarAxisLabel:nth-of-type(2){fill:#808A94!important}
    .radarAxisLabel:nth-of-type(3){fill:#808A94!important}
    .radarAxisLabel:nth-of-type(4){fill:#808A94!important}
    .radarAxisLabel:nth-of-type(5){fill:#808A94!important}
    .radarAxisLabel:nth-of-type(6){fill:#808A94!important}
    .radarAxisLabel:nth-of-type(7){fill:#808A94!important}
  themeVariables:
    cScale0: "#62B5E5"
    cScale1: "#62B5E5"
    cScale2: "#62B5E5"
    cScale3: "#62B5E5"
    cScale4: "#F97316"
    cScale5: "#F97316"
    cScale6: "#F97316"
    radar:
      axisColor: "#9CA3AF"
      graticuleColor: "#E5E7EB"
      graticuleOpacity: 0
      axisStrokeWidth: 1
      curveOpacity: 0
---
radar-beta
  axis b0["Basic Blocking Locust 100ms: 99 ms ▬ 0%"]
  axis b1["Basic Blocking Locust 20ms: 19 ms ▬ 0%"]
  axis b2["Basic Blocking Locust 2ms: 9 ms ▬ 0%"]
  axis b3["Commit Latency 16ms: 4 ms ▬ -16%"]
  axis b4["Commit Latency 1ms: 1.94 ms ▬ 0%"]
  axis b5["Commit Latency 256ms: 214 ms ▬ +1%"]
  axis b6["Logging JWT Blocking Locust: 20 ms ▬ 0%"]
  curve stddev2_high["main EWMA + 2 std dev"]{100.00, 100.00, 108.28, 151.61, 104.00, 104.46, 100.00}
  curve stddev1_high["main EWMA + 1 std dev"]{100.00, 100.00, 104.14, 125.81, 102.00, 102.23, 100.00}
  curve stddev1_low["main EWMA - 1 std dev"]{100.00, 100.00, 95.86, 74.19, 98.00, 97.77, 100.00}
  curve stddev2_low["main EWMA - 2 std dev"]{100.00, 100.00, 91.72, 48.39, 96.00, 95.54, 100.00}
  curve branch_2["#8201 (2 runs earlier)"]{100.00, 100.00, 89.20, 80.42, 102.19, 101.22, 100.00}
  curve branch_3["#8201 (1 run earlier)"]{100.00, 100.00, 100.35, 86.67, 102.55, 101.00, 100.00}
  curve branch_4["#8201"]{100.00, 100.00, 100.35, 83.86, 100.36, 100.66, 100.00}
  graticule polygon
  max 188
  min 12
  ticks 0
  showLegend false
Loading

Memory (bytes)

---
config:
  radar:
    width: 620
    height: 620
    marginTop: 90
    marginRight: 220
    marginBottom: 60
    marginLeft: 220
    axisLabelFactor: 1.12
    curveTension: 0.08
  theme: base
  themeCSS: |
    .radarCurve-0{fill:color-mix(in srgb, #62B5E5 13%, var(--color-canvas-default,var(--bgColor-default,#fff)))!important;fill-opacity:1!important;stroke:none!important;stroke-width:0!important}
    .radarCurve-1{fill:color-mix(in srgb, #62B5E5 40%, var(--color-canvas-default,var(--bgColor-default,#fff)))!important;fill-opacity:1!important;stroke:none!important;stroke-width:0!important}
    .radarCurve-2{fill:color-mix(in srgb, #62B5E5 13%, var(--color-canvas-default,var(--bgColor-default,#fff)))!important;fill-opacity:1!important;stroke:none!important;stroke-width:0!important}
    .radarCurve-3{fill:var(--color-canvas-default,var(--bgColor-default,#fff))!important;fill-opacity:1!important;stroke:none!important;stroke-width:0!important}
    .radarAxisLabel,.radarTitle{fill:var(--color-fg-default,var(--fgColor-default,#111827))!important;color:var(--color-fg-default,var(--fgColor-default,#111827))!important}
    .radarCurve-4{stroke-width:1.5px!important;stroke-opacity:0.40!important}
    .radarCurve-5{stroke-width:1.5px!important;stroke-opacity:0.50!important}
    .radarCurve-6{stroke-width:1.75px!important;stroke-opacity:1.00!important}
    .radarAxisLabel:nth-of-type(1){fill:#2DA44E!important}
    .radarAxisLabel:nth-of-type(2){fill:#808A94!important}
    .radarAxisLabel:nth-of-type(3){fill:#E5484D!important}
    .radarAxisLabel:nth-of-type(4){fill:#808A94!important}
    .radarAxisLabel:nth-of-type(5){fill:#E5484D!important}
    .radarAxisLabel:nth-of-type(6){fill:#808A94!important}
    .radarAxisLabel:nth-of-type(7){fill:#808A94!important}
    .radarAxisLabel:nth-of-type(8){fill:#E5484D!important}
  themeVariables:
    cScale0: "#62B5E5"
    cScale1: "#62B5E5"
    cScale2: "#62B5E5"
    cScale3: "#62B5E5"
    cScale4: "#F97316"
    cScale5: "#F97316"
    cScale6: "#F97316"
    radar:
      axisColor: "#9CA3AF"
      graticuleColor: "#E5E7EB"
      graticuleOpacity: 0
      axisStrokeWidth: 1
      curveOpacity: 0
---
radar-beta
  axis b0["Basic: 84.9 MiB ▼ 3%"]
  axis b2["Basic Blocking Locust 100ms: 89.7 MiB ▬ -1%"]
  axis b3["Basic Blocking Locust 20ms: 91.8 MiB ▲ 1%"]
  axis b4["Basic Blocking Locust 2ms: 93.8 MiB ▬ 0%"]
  axis b5["Basic JS: 74.8 MiB ▲ 6%"]
  axis b6["Basic Multi-Threaded: 88.8 MiB ▬ 0%"]
  axis b7["Logging: 75.9 MiB ▬ -1%"]
  axis b9["Logging JWT Blocking Locust: 90.4 MiB ▲ 1%"]
  curve stddev2_high["main EWMA + 2 std dev"]{105.12, 102.24, 101.99, 101.78, 104.02, 102.75, 101.42, 101.24}
  curve stddev1_high["main EWMA + 1 std dev"]{102.56, 101.12, 101.00, 100.89, 102.01, 101.38, 100.71, 100.62}
  curve stddev1_low["main EWMA - 1 std dev"]{97.44, 98.88, 99.00, 99.11, 97.99, 98.62, 99.29, 99.38}
  curve stddev2_low["main EWMA - 2 std dev"]{94.88, 97.76, 98.01, 98.22, 95.98, 97.25, 98.58, 98.76}
  curve branch_2["#8201 (2 runs earlier)"]{96.53, 100.00, 101.69, 99.50, 100.15, 101.30, 98.30, 100.00}
  curve branch_3["#8201 (1 run earlier)"]{99.34, 100.54, 98.93, 100.87, 99.07, 100.71, 101.02, 101.51}
  curve branch_4["#8201"]{96.78, 99.06, 101.07, 100.35, 105.64, 100.09, 99.42, 100.73}
  graticule polygon
  max 110
  min 90
  ticks 0
  showLegend false
Loading

Rate (ops/s)

---
config:
  radar:
    width: 620
    height: 620
    marginTop: 90
    marginRight: 220
    marginBottom: 60
    marginLeft: 220
    axisLabelFactor: 1.12
    curveTension: 0.08
  theme: base
  themeCSS: |
    .radarCurve-0{fill:color-mix(in srgb, #62B5E5 13%, var(--color-canvas-default,var(--bgColor-default,#fff)))!important;fill-opacity:1!important;stroke:none!important;stroke-width:0!important}
    .radarCurve-1{fill:color-mix(in srgb, #62B5E5 40%, var(--color-canvas-default,var(--bgColor-default,#fff)))!important;fill-opacity:1!important;stroke:none!important;stroke-width:0!important}
    .radarCurve-2{fill:color-mix(in srgb, #62B5E5 13%, var(--color-canvas-default,var(--bgColor-default,#fff)))!important;fill-opacity:1!important;stroke:none!important;stroke-width:0!important}
    .radarCurve-3{fill:var(--color-canvas-default,var(--bgColor-default,#fff))!important;fill-opacity:1!important;stroke:none!important;stroke-width:0!important}
    .radarAxisLabel,.radarTitle{fill:var(--color-fg-default,var(--fgColor-default,#111827))!important;color:var(--color-fg-default,var(--fgColor-default,#111827))!important}
    .radarCurve-4{stroke-width:1.5px!important;stroke-opacity:0.20!important}
    .radarCurve-5{stroke-width:1.5px!important;stroke-opacity:0.30!important}
    .radarCurve-6{stroke-width:1.5px!important;stroke-opacity:0.40!important}
    .radarCurve-7{stroke-width:1.5px!important;stroke-opacity:0.50!important}
    .radarCurve-8{stroke-width:1.75px!important;stroke-opacity:1.00!important}
    .radarAxisLabel:nth-of-type(1){fill:#808A94!important}
    .radarAxisLabel:nth-of-type(2){fill:#808A94!important}
    .radarAxisLabel:nth-of-type(3){fill:#808A94!important}
    .radarAxisLabel:nth-of-type(4){fill:#808A94!important}
    .radarAxisLabel:nth-of-type(5){fill:#808A94!important}
    .radarAxisLabel:nth-of-type(6){fill:#E5484D!important}
  themeVariables:
    cScale0: "#62B5E5"
    cScale1: "#62B5E5"
    cScale2: "#62B5E5"
    cScale3: "#62B5E5"
    cScale4: "#F97316"
    cScale5: "#F97316"
    cScale6: "#F97316"
    cScale7: "#F97316"
    cScale8: "#F97316"
    radar:
      axisColor: "#9CA3AF"
      graticuleColor: "#E5E7EB"
      graticuleOpacity: 0
      axisStrokeWidth: 1
      curveOpacity: 0
---
radar-beta
  axis b0["CHAMP get: 37,674,068 ops/s ▬ 0%"]
  axis b1["CHAMP put: 5,382,151 ops/s ▬ -1%"]
  axis b2["KV deserialisation: 1,639,613 ops/s ▬ +1%"]
  axis b3["KV serialisation: 1,468,213 ops/s ▬ +2%"]
  axis b4["KV s…t deserialisation: 4,112 ops/s ▬ -1%"]
  axis b5["KV snapshot serialisation: 3,976 ops/s ▼ 12%"]
  curve stddev2_high["main EWMA + 2 std dev"]{105.60, 106.13, 106.18, 107.03, 105.57, 111.84}
  curve stddev1_high["main EWMA + 1 std dev"]{102.80, 103.07, 103.09, 103.51, 102.78, 105.92}
  curve stddev1_low["main EWMA - 1 std dev"]{97.20, 96.93, 96.91, 96.49, 97.22, 94.08}
  curve stddev2_low["main EWMA - 2 std dev"]{94.40, 93.87, 93.82, 92.97, 94.43, 88.16}
  curve branch_0["#8201 (4 runs earlier)"]{100.54, 99.17, 99.84, 98.97, 100.77, 103.24}
  curve branch_1["#8201 (3 runs earlier)"]{101.03, 101.47, 99.84, 99.24, 101.13, 104.60}
  curve branch_2["#8201 (2 runs earlier)"]{100.99, 99.42, 100.33, 100.99, 101.83, 96.03}
  curve branch_3["#8201 (1 run earlier)"]{101.01, 100.76, 100.33, 100.68, 100.32, 100.34}
  curve branch_4["#8201"]{99.76, 98.68, 101.48, 101.86, 99.49, 88.08}
  graticule polygon
  max 121
  min 79
  ticks 0
  showLegend false
Loading

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
@achamayou
Amaury Chamayou (achamayou) marked this pull request as ready for review August 24, 2026 20:52
Comment thread tests/infra/locust_benchmark.py
@achamayou
Amaury Chamayou (achamayou) merged commit f3c2b0f into main Aug 25, 2026
16 checks passed
@achamayou
Amaury Chamayou (achamayou) deleted the achamayou-logging-jwt-locust-benchmark branch August 25, 2026 14:57
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants