LoadBalancerPro is a Java 17 / Spring Boot reverse proxy plus a controlled, opt-in enterprise lab. The default deployable artifact contains the real proxy runtime and its operator status surface; calculation, simulation, experiment evidence, demos, and guarded cloud-management tools are isolated in a Maven lab profile.
The default posture is conservative: API-key authentication is selected, proxying and LASE shadow mode are disabled, cloud mutation is dry-run, telemetry export is off, and only health/info Actuator endpoints are exposed.
- Opt-in reverse proxy data plane with configured routes, bounded request size and timeout, live per-upstream routing telemetry, exact rolling tail latency, active health checks, retry budgets/backoff, cooldown/slow-start recovery, process-local concurrency/load-shedding controls, status, recent decisions, and guarded reload.
- Production-boundary executable Spring Boot JAR, non-root proxy Docker image, hardened Compose deployment, CycloneDX SBOM generation, and load smoke/soak helpers.
- Fail-closed reviewed-staging qualification binding prior/candidate registry digests, observed deployment state, and loaded rollout/rollback, plus a deployment-equivalent capacity staircase bound to the exact candidate and per-replica telemetry; CI-gated Compose and live two-zone Kubernetes proofs cover distribution, candidate abort, content-distinct image rollout and baseline rollback under load, immutable inbound-TLS identity rotation and rollback, bounded API-key overlap/commit/rollback, pod-identity turnover, endpoint continuity, replica loss, planned worker removal, operator-remediated abrupt worker loss, degraded service, and worker recovery.
- API-key mode with a required primary key and at most one optional rotation key, plus OAuth2 resource-server mode, with deny-by-default API classification.
- Actuator health/readiness, optional Prometheus metrics, and optional OTLP metrics export with endpoint validation.
- Capacity-aware, predictive, and evaluation-only allocation APIs in the separate source/Lab Tools runtime.
- Deterministic request-level comparison, Decision Explorer/browser cockpit surfaces, Enterprise Lab scenarios, durable evidence, and packaged proof tools in the opt-in Lab Tools artifact.
- A guarded AWS
CloudManagerboundary using AWS SDK v2; live mutation requires explicit operator and account/region/capacity gates.
The live /proxy/** data plane and production-only artifact are built. The project is now qualifying an exact release
for forecast traffic. Work in this phase is load-balancer engineering and deployment proof:
- Lock the expected request rate, concurrency, route and payload mix, upstream behavior, failure cases, and objectives.
- Run a repeatable capacity staircase on deployment-equivalent resources and locate the first saturation point.
- Prove TLS, authentication, target policy, observability, reload, drain, restart, and certificate rotation in a reviewed staging boundary.
- Choose single-active or multiple-active topology and verify its failover, configuration, connection, and state behavior.
- Promote an immutable image through controlled traffic steps with tested abort and rollback thresholds.
LOAD_BALANCER_BUILD_OUT.md is the execution plan and readiness gate for this phase.
The local capacity, staging, deployment-capacity, and active-active/image-replacement runners now exist. The
deployment-capacity lane binds the reviewed forecast to the exact staging-profile hash and candidate digest, requires
fresh replica identities and hash-pinned per-replica telemetry at every repeat, recomputes the saturation envelope,
and restores the prior digest. The Kubernetes adapter compiler supplies rollout, rollback, fault, reset, restart,
certificate-rotation, deployment-inspection, and capacity-sampling executables. A disposable two-worker/two-zone kind
lane now deploys the restricted production image and proves live Service distribution, a metadata-only content-distinct
candidate rollout and baseline rollback, then rotates between independently rooted certificates through versioned
immutable Secrets and restores the original identity. It then rolls through immutable A-only, A+B, and B-only API-key
Secrets and reverses that sequence for rollback. Every credential transition preserves two ready endpoints, turns over
both pod UIDs, keeps the runtime image fixed, and proves the retired key is rejected before the lane exercises worker
drain/stop, degraded traffic, and
operator-remediated no-drain worker loss and recovery. The next action remains to compile the adapters from the reviewed
staging
cluster identity, freeze the observed configuration/ingress hashes into the profiles, then run staging qualification
and the capacity staircase against that exact candidate.
Local and CI results do not establish deployment capacity, release compatibility, registry integrity,
production-ingress behavior, or automatic infrastructure-failure detection.
Authentication, TLS verification, secret handling, bounded concurrency, dependency/image scanning, and protected merge gates remain mandatory while this work advances.
- Java 17 or later
- Maven 3.9 or later
- Docker, optional
- PowerShell or Bash for the supplied smoke helpers
Run the local profile:
mvn spring-boot:run "-Dspring-boot.run.arguments=--server.address=127.0.0.1 --spring.profiles.active=local"Verify the service:
curl -fsS http://127.0.0.1:8080/api/healthOpen:
Build and run the executable JAR:
mvn -B package
java -jar "$(bash scripts/resolve-executable-jar.sh)" \
--server.address=127.0.0.1 \
--server.port=18080 \
--spring.profiles.active=localThis default JAR is the production proxy artifact. It serves /proxy-status.html and omits lab, CLI, demo, GUI, Decision Explorer/replay/evidence-training services, ServerMonitor, and lab-only dependencies. Build simulation and CLI tooling explicitly:
mvn -B -P lab -DskipTests package
java -jar "$(bash scripts/resolve-executable-jar.sh --lab)" --lase-demo=healthyPowerShell resolves the same Maven project.build.finalName:
$jar = & .\scripts\resolve-executable-jar.ps1
java -jar $jar --server.address=127.0.0.1 --server.port=18080 --spring.profiles.active=local| Profile | Intended use | Authentication | Proxy |
|---|---|---|---|
| default | Protected configuration baseline | API key; startup fails without a configured key | Disabled |
local |
Loopback development and browser review | Disabled with a startup warning | Disabled |
prod |
Protected deployment baseline | API key from LOADBALANCERPRO_API_KEY |
Disabled |
cloud-sandbox |
Guarded sandbox configuration | API key; cloud dry-run and mutation disabled | Disabled |
| OAuth2 configuration | Trusted issuer/JWK deployment | JWT roles | Disabled unless explicitly enabled |
proxy-demo-* |
Scripted loopback proxy fixtures | Disabled; loopback only | Explicitly enabled |
cloud-sandbox and proxy-demo-* are source/Lab Tools profiles and are not packaged in the default production JAR.
Copyable profile examples are under docs/examples/operator-run-profiles. See OPERATOR_RUN_PROFILES.md for the full run matrix.
Important defaults in application.properties:
| Property | Default | Effect |
|---|---|---|
loadbalancerpro.auth.mode |
api-key |
Protected API mode |
loadbalancerpro.api.rotation-key |
empty | Optional second key accepted only during an operator-bounded rotation overlap |
loadbalancerpro.proxy.enabled |
false |
No forwarding until explicitly enabled |
loadbalancerpro.lase.shadow.enabled |
false |
No shadow evaluation by default |
loadbalancerpro.api.max-request-bytes |
16384 |
Bounded API request bodies |
loadbalancerpro.proxy.max-request-bytes |
65536 |
Bounded proxied request bodies |
loadbalancerpro.proxy.connect-timeout |
1s |
Bounded upstream connection establishment |
loadbalancerpro.proxy.request-timeout |
2s |
Bounded upstream request |
loadbalancerpro.proxy.routes.<name>.request-timeout |
inherits global | Per-route upstream request bound |
loadbalancerpro.proxy.routes.<name>.hash-on |
client-ip |
Consistent-hash key source; immediate peer address or an operator-configured header that requires a trusted ingress boundary |
loadbalancerpro.proxy.routes.<name>.affinity.cookie-name |
empty | Enables route-local signed routing affinity only when paired with an operator-supplied HMAC key |
loadbalancerpro.proxy.forwarded.mode |
strip-and-set |
Replace caller-supplied forwarding metadata by default |
loadbalancerpro.proxy.limits.max-in-flight |
0 |
Process-local global cap; 0 leaves it unlimited |
loadbalancerpro.proxy.limits.adaptive |
false |
Opt-in latency-feedback adjustment below the configured cap |
loadbalancerpro.proxy.shedding.enabled |
false |
Opt-in priority-aware process-local load shedding |
loadbalancerpro.proxy.health-check.enabled |
false |
Opt-in active checks |
loadbalancerpro.proxy.health-check.healthy-threshold |
2 |
Successful background probes required to recover |
loadbalancerpro.proxy.health-check.unhealthy-threshold |
3 |
Failed background probes required to mark unhealthy |
loadbalancerpro.proxy.retry.enabled |
false |
Opt-in retries |
loadbalancerpro.proxy.retry.budget-percent |
20 |
Process-local retry credits per 100 admitted primary requests |
loadbalancerpro.proxy.retry.backoff.base |
50ms |
Initial full-jitter exponential backoff ceiling |
loadbalancerpro.proxy.retry.backoff.max |
1s |
Maximum retry backoff ceiling |
loadbalancerpro.proxy.retry.retry-non-idempotent |
false |
No default non-idempotent retry |
loadbalancerpro.proxy.cooldown.enabled |
false |
Opt-in backend cooldown |
loadbalancerpro.proxy.slow-start.duration |
0s |
Opt-in linear effective-weight ramp for new/recovered upstreams |
management.prometheus.metrics.export.enabled |
false |
No default Prometheus export |
management.otlp.metrics.export.enabled |
false |
No default OTLP export |
management.endpoints.web.exposure.include |
health,info |
Minimal Actuator exposure |
In the source/Lab Tools API runtime, GET /api/health and unauthenticated OPTIONS are the public API exceptions. Other /api/** routes, /proxy/**, OpenAPI, and Swagger require the configured key. The production artifact uses /actuator/health for health checks. Use:
export LOADBALANCERPRO_API_KEY='supply-from-a-secret-manager'
java -jar "$(bash scripts/resolve-executable-jar.sh)" --spring.profiles.active=prodAPI-key rotation is deliberately bounded to two credentials. Roll out a configuration containing primary A plus rotation key B, switch clients to B, then roll out B as the sole primary. Rollback reverses the sequence through the same A+B overlap. A rotation key cannot replace a missing primary, and the process does not dynamically reread mounted credential files.
Do not commit API keys, OAuth tokens, AWS credentials, telemetry headers, private keys, or production targets. Terminate TLS at a trusted reverse proxy, ingress, managed load balancer, platform edge, or service mesh before shared-network exposure.
OTLP metrics are opt-in. When enabled, the endpoint validator rejects blank or malformed URLs, embedded credentials, query strings, fragments, disallowed localhost, and obvious public hosts when private endpoints are required:
management.otlp.metrics.export.enabled=true
management.otlp.metrics.export.url=http://localhost:4318/v1/metrics
loadbalancerpro.telemetry.otlp.require-private-endpoint=trueProxy mode requires explicit routes or upstreams. Start with the loopback smoke instead of adapting production targets directly:
pwsh ./scripts/smoke/operator-run-profiles-smoke.ps1 -PackageThe production proxy exposes forwarding under /proxy/**, read-only state at GET /api/proxy/status, the newest 100 process-local forwarding decisions at GET /api/proxy/decisions/recent, read-only analysis for a retained attempt at GET /api/proxy/decisions/{decisionId}/explain, and guarded configuration reload at POST /api/proxy/reload. Named routes support exact host/header predicates and deterministic percentage split groups in addition to path prefixes. The explanation uses score and factor evidence captured with the actual selection; it does not rerun a stateful, sampled, keyed, positional, or affinity strategy later. Opt-in LASE shadow evaluation and GET /api/lase/shadow remain available only in the source/Lab Tools runtime. Operator configuration examples are under docs/examples/proxy.
Use these as the forecast-load qualification and proxy-operation path:
LOAD_BALANCER_BUILD_OUT.mdREVERSE_PROXY_MODE.mdLIVE_PROXY_CONTAINMENT.mdPRIVATE_NETWORK_PROXY_PROFILE_PLAN.mdAPI_SECURITY.md
| Surface | Purpose |
|---|---|
GET /api/health |
Lightweight application health |
POST /api/allocate/capacity-aware |
Capacity-aware calculation |
POST /api/allocate/predictive |
Predictive calculation |
POST /api/allocate/evaluate |
Allocation evaluation with optional LASE summaries |
POST /api/routing/compare |
Read-only strategy comparison |
POST /api/routing/decision-explorer |
Compact routing explanation |
/api/lab/** |
Controlled lab scenarios, decisions, runs, policy, metrics, and experiments |
/api/enterprise-lab/** |
Reviewer summaries and evidence views |
/proxy/** |
Optional HTTP forwarding |
GET /api/proxy/status |
Proxy configuration, health, and live per-upstream runtime statistics |
GET /api/proxy/decisions/recent |
Bounded process-local records of actual upstream forwarding attempts |
GET /api/proxy/decisions/{decisionId}/explain |
Dominant-factor, delta, counterfactual, and score-tie analysis from one retained actual forwarding attempt |
GET /api/lase/shadow |
Bounded process-local allocation and opt-in live-proxy shadow observations plus dispatch counters |
POST /api/proxy/reload |
Guarded proxy configuration reload |
/actuator/health |
Spring Boot health and readiness |
Request and response contracts live in API_CONTRACTS.md. Generated OpenAPI is available at /v3/api-docs when permitted by the selected auth mode.
Run tests, build the executable artifact, or generate the JaCoCo verification report:
mvn -B test
mvn -B package
mvn -B verifyRun the bounded local proxy regression scenarios (Docker Compose, Bash, curl, jq, OpenSSL, and Vegeta required):
bash scripts/bench/proxy-benchmark-soak.sh --mode smokeThe smoke and scheduled one-hour soak gates use only the TLS-authenticated loopback Compose stack and ignored
target/bench/ output. See scripts/bench/README.md for scenarios, thresholds, and the
evidence boundary; local results do not establish production SLOs, capacity, public-ingress safety, or p95/p99
guarantees.
Generate CycloneDX JSON and XML SBOMs:
mvn -B org.cyclonedx:cyclonedx-maven-plugin:2.9.1:makeAggregateBom \
-DoutputFormat=all \
-DoutputDirectory=target \
-DoutputName=bom \
-DincludeProvidedScope=false \
-Dcyclonedx.skipAttach=trueInspect the packaged artifact:
bash scripts/local-artifact-verify.sh --build
bash scripts/operator-distribution-smoke.sh --package --run-jar-smokePowerShell equivalents:
pwsh ./scripts/local-artifact-verify.ps1 -Build
pwsh ./scripts/operator-distribution-smoke.ps1 -Package -RunJarSmokeBuild and smoke the protected container:
docker build -t loadbalancerpro:local .
docker run --rm --name loadbalancerpro-local \
-p 127.0.0.1:8080:8080 \
-e LOADBALANCERPRO_API_KEY=CHANGE_ME_LOCAL_API_KEY \
loadbalancerpro:local
curl -fsS http://127.0.0.1:8080/actuator/healthThe Docker image runs as a non-root user and defaults to the prod profile. Keep the host port loopback-bound for local checks. See CONTAINER_DEPLOYMENT.md for networking and TLS boundaries.
- Record the workload contract and acceptance thresholds from the build-out.
- Build and verify the exact production artifact with its SBOM and image scans.
- Supply TLS, authentication, trust, identity, upstream, and monitoring configuration through deployment secret and configuration management.
- Pass the loopback regression smoke and one-hour soak without weakening verification or safety limits.
- Run the capacity staircase on deployment-equivalent resources and select an operating envelope with agreed headroom below saturation.
- Repeat the forecast mix and failure cases in the reviewed staging boundary.
- Prove the selected single-active or multiple-active topology, including drain, replacement, and configuration consistency behavior.
- Roll out an immutable image in controlled traffic steps; abort and restore the prior image/configuration when an agreed threshold is exceeded.
For the packaged-application proof path use DEPLOYMENT_SMOKE_KIT.md. For controlled lab tooling use LOCAL_LAB_MANUAL_TOOLING_INDEX.md.
- Startup says the API key is missing: set
LOADBALANCERPRO_API_KEYforprod/default API-key mode, or explicitly uselocalonly on loopback. - The executable JAR cannot be found: run
mvn -B package, then use the resolver script; do not hard-code an artifact filename. - A protected route returns 401: include
X-API-Keyor configure the intended OAuth2 issuer, JWK source, and roles. - Proxy status says disabled: enable proxy mode only through a reviewed profile and provide valid routes/upstreams.
- An upstream is skipped or retry is suppressed: inspect health, retry-budget counters, cooldown, slow-start weight, timeout, and route status in
/api/proxy/statusand application logs. - OTLP startup validation fails: use a trusted private/internal collector URL without credentials, query parameters, or fragments.
- Docker cannot reach a host backend:
127.0.0.1inside a container is the container; use an explicit Docker network or platform host gateway. - A port is already in use: select another
server.portand keep the bind address explicit. - Local Maven trust errors occur: repair the workstation/JDK trust store; do not disable TLS verification.