feat: DH-21975: Gc benchmarking

[stanbrub]

Summary

GC benchmarking infrastructure and analysis for understanding how each garbage collector (G1, ZGC, Shenandoah, Parallel) uses heap across Deephaven operation categories.

Infrastructure:

• Matrix workflow framework for running heap-pressure benchmarks at minimal viable heap per GC/operation combination
• Benchmark event collection: GC activity (via JFR), OS metrics (RSS, CPU), and UGP cycle costs (via update listeners)
• Local Parquet generator as alternative to Kafka-based data loading
• Train test suite covering 6 operation categories (Filter, AggBy, UserFormula, NaturalJoin, Ordered, UpdateBy)
• Train dashboard for visualizing results
• Improved robustness of test server setup and data sharing between jobs

Analysis (docs/):

• docs/gc/gc-report.md — GC throughput rankings from the original 48G-heap training benchmarks (static + incremental, JDK 17/25)
• docs/heap/gc-heap.md — Heap-pressure study showing how each GC behaves when constrained to minimal heap per operation category.

[cpwright]

All the benchmarks are going to do work. I have some concerns that we might do too much work related to the timestamp calculation and traversing a UnionSourceManager (merge) where not necessary.

[Copilot]

Pull request overview

Adds GC benchmarking infrastructure, training workloads, event collection, dashboards, heap-pressure analysis, and more robust remote execution.

Changes:

• Adds local Parquet generation and six-category training benchmarks.
• Adds GC/heap dashboards, reports, and consolidated data tooling.
• Adds matrix workflows and remote-server lifecycle improvements.

Reviewed changes

Copilot reviewed 56 out of 72 changed files in this pull request and generated 8 comments.

Show a summary per file

File	Description
src/test/resources/io/deephaven/benchmark/util/filerfolder/folder1/filer2.txt	Adds directory-size fixture.
src/test/resources/io/deephaven/benchmark/util/filerfolder/filer1.txt	Adds directory-size fixture.
src/test/java/io/deephaven/benchmark/util/FilerTest.java	Tests byte-size calculation.
src/test/java/io/deephaven/benchmark/api/QueryLogTest.java	Updates query-log expectations.
src/main/resources/io/deephaven/benchmark/run/profile/queries/dashboards/train_dashboard.dh.py	Adds GC training dashboards.
src/main/resources/io/deephaven/benchmark/run/profile/queries/dashboards/pullup.py	Consolidates benchmark datasets.
src/main/resources/io/deephaven/benchmark/run/profile/queries/dashboards/heap_dashboard.dh.py	Adds heap-pressure dashboard.
src/main/resources/io/deephaven/benchmark/run/profile/queries/dashboards/benchmark_functions.dh.py	Adds event and consolidated-file loading.
src/main/resources/io/deephaven/benchmark/run/profile/queries/compare.py	Updates product comparisons.
src/main/java/io/deephaven/benchmark/util/Filer.java	Adds file creation and sizing utilities.
src/main/java/io/deephaven/benchmark/generator/LocalParquetGenerator.java	Adds direct Parquet generation.
src/main/java/io/deephaven/benchmark/generator/ColumnDefs.java	Adds copying and larger caches.
src/main/java/io/deephaven/benchmark/generator/AvroKafkaGenerator.java	Strengthens Kafka delivery settings.
src/main/java/io/deephaven/benchmark/controller/DeephavenDockerController.java	Extends startup timeout.
src/main/java/io/deephaven/benchmark/connect/BarrageConnector.java	Improves cleanup and message limits.
src/main/java/io/deephaven/benchmark/api/Snippets.java	Adds event tables and Parquet links.
src/main/java/io/deephaven/benchmark/api/QueryLog.java	Simplifies query-log placement.
src/main/java/io/deephaven/benchmark/api/BenchTable.java	Integrates local Parquet datasets.
src/main/java/io/deephaven/benchmark/api/BenchQuery.java	Cleans query-scope variables.
src/main/java/io/deephaven/benchmark/api/BenchPlatform.java	Corrects JVM argument collection.
src/main/java/io/deephaven/benchmark/api/BenchLog.java	Relocates engine logs.
src/main/java/io/deephaven/benchmark/api/BenchEvents.java	Persists benchmark events.
src/main/java/io/deephaven/benchmark/api/Bench.java	Integrates events and per-test logs.
src/it/java/io/deephaven/benchmark/tests/train/UserFormulaTrainTest.java	Adds UDF training workload.
src/it/java/io/deephaven/benchmark/tests/train/UpdateByTrainTest.java	Adds UpdateBy workload.
src/it/java/io/deephaven/benchmark/tests/train/TrainTestRunner.java	Adds training orchestration and telemetry.
src/it/java/io/deephaven/benchmark/tests/train/OrderedTrainTest.java	Adds ordered aggregation workload.
src/it/java/io/deephaven/benchmark/tests/train/NaturalJoinTrainTest.java	Adds natural-join workload.
src/it/java/io/deephaven/benchmark/tests/train/InlineFormulaTrainTest.java	Adds inline-formula workload.
src/it/java/io/deephaven/benchmark/tests/train/FilterTrainTest.java	Adds filter workload.
src/it/java/io/deephaven/benchmark/tests/train/AggByTrainTest.java	Adds aggregation workload.
src/it/java/io/deephaven/benchmark/tests/standard/StandardTestRunner.java	Adds local loading and event capture.
src/it/java/io/deephaven/benchmark/tests/standard/kafka/KafkaTestRunner.java	Configures log package roots.
src/it/java/io/deephaven/benchmark/tests/standard/file/FileTestRunner.java	Configures log package roots.
src/it/java/io/deephaven/benchmark/tests/compare/CompareTestRunner.java	Configures log package roots.
pom.xml	Adds Parquet writer dependency.
docs/heap/gc-heap.md	Documents heap-pressure findings.
docs/gc/train-dashboard.md	Documents dashboard setup.
docs/gc/gc-report.md	Documents GC throughput findings.
.github/workflows/remote-benchmarks.yml	Improves remote setup and publishing.
.github/workflows/matrix-exist-remote-benchmarks.yml	Adds script-generated matrices.
.github/workflows/adhoc-exist-remote-benchmarks.yml	Clears existing servers first.
.github/workflows/adhoc-auto-remote-benchmarks.yml	Clears between benchmark runs.
.github/scripts/setup-test-server-remote.sh	Reuses repository checkouts.
.github/scripts/run-ssh-local.sh	Adds SSH retries.
.github/scripts/run-benchmarks-remote.sh	Caps runner heap.
.github/scripts/manage-deephaven-remote.sh	Improves image selection and host paths.
.github/scripts/clear-test-server-remote.sh	Adds remote cleanup.
.github/scripts/build-server-distribution-remote.sh	Reuses server builds.
.github/scripts/build-matrix.sh	Converts TSV profiles to matrices.
.github/scripts/build-docker-image-remote.sh	Reuses Docker images.
.github/scripts/build-benchmark-artifact-remote.sh	Reuses benchmark artifacts.
.github/scripts/adhoc.sh	Improves workflow error reporting.
.github/resources/adhoc-benchmark-docker-compose.yml	Exposes the host data directory.
.github/matrix/gc-train-heap-fail.sh	Defines heap-pressure runs.
.github/matrix/gc-train-1000-100.sh	Defines GC comparison runs.

Suppressed comments (2)

.github/matrix/gc-train-1000-100.sh:25

• train.staticInc has no consumer in the repository; TrainTestRunner runs both modes whenever its configured factors are positive. Thus rows marked Inc still run and publish Static benchmarks. Wire this setting into mode selection or remove the unused axis and adjust the matrix scope.

  local opts="$2 $BASE -DPeriodicUpdateGraph.targetCycleDurationMillis=$3 -Dtrain.incLoadTarget=$4 -Dtrain.staticInc=$5"

src/main/resources/io/deephaven/benchmark/run/profile/queries/dashboards/train_dashboard.dh.py:134

• The new event pipeline never emits server_state_log / ugp.cycle.time; it emits ugp.cycle.cost / duration_rows, with cycle time stored in duration. As written, the on-budget table has no rows for these benchmark runs. Align this filter/value column with TrainTestRunner or change the producer schema.

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[Copilot]

Pull request overview

Copilot reviewed 57 out of 73 changed files in this pull request and generated 1 comment.

Suppressed comments (2)

src/main/resources/io/deephaven/benchmark/run/profile/queries/dashboards/benchmark_functions.dh.py:179

• The generic load_table_or_empty() fallback resolves empty_bench_{table_name}, but the new events table has no empty_bench_events() definition. Calling load_table_or_empty('events', ...) without both filters therefore raises NameError instead of returning an empty table. Add the matching empty schema, including set_id and run_id, as provided for results, metrics, and platform.
  src/it/java/io/deephaven/benchmark/tests/standard/StandardTestRunner.java:299
• Static runs never trim standard_events to the measured interval. JFR starts before bench_api_metrics_start(), whose forced garbage_collect() occurs before begin_clock, so every static GC analysis includes at least that setup GC (and process samples outside the operation). Apply the same begin/end filter already used by the incremental query after teardown.

[Copilot]

Pull request overview

Copilot reviewed 57 out of 73 changed files in this pull request and generated 8 comments.

[Copilot]

Pull request overview

Copilot reviewed 57 out of 73 changed files in this pull request and generated 1 comment.

Suppressed comments (6)

Previously missed (6) — in code that hasn't changed since the last review.

src/main/java/io/deephaven/benchmark/generator/LocalParquetGenerator.java:52

• The local path retains the requested compression in the table definition/cache key, but this writer is created without that codec and therefore uses parquet-floor's default. Profiles set codecs such as LZO, so training files are not written with the advertised configuration. Pass the selected codec into this generator and configure the writer accordingly (with an explicit mapping or validation for unsupported codecs).
  src/main/resources/io/deephaven/benchmark/run/profile/queries/dashboards/train_dashboard.dh.py:26
• benchmark_functions2.dh.py is not present anywhere in the repository; the helper updated by this PR is benchmark_functions.dh.py. This dashboard therefore fails in a checkout/NFS deployment unless an unrelated manually published object happens to exist. Load the tracked helper instead.
  src/main/resources/io/deephaven/benchmark/run/profile/queries/dashboards/heap_dashboard.dh.py:15
• benchmark_functions2.dh.py is not present anywhere in the repository; the helper updated by this PR is benchmark_functions.dh.py. This dashboard therefore fails in a checkout/NFS deployment unless an unrelated manually published object happens to exist. Load the tracked helper instead.
  src/main/java/io/deephaven/benchmark/api/BenchEvents.java:30
• The documented event schema names a nonexistent detail column and omits the required name and value columns used below. Correct the contract so callers can construct a compatible ResultTable.
  docs/heap/gc-heap.md:14
• Correct the grammatical typo.
  docs/gc/train-dashboard.md:3
• Correct the grammatical typo.

The training dashboard can be used to view the data collected from the GC effort so far. It is structure to be accessed via a python snippet that loads the dashboard and data from a GCloud bucket. This only works for a DHC docker image, or a native DHC/Python install that has a writable "/data" directory.