feat: add bounded local capacity telemetry

docs/CAPACITY-TELEMETRY.md

@@ -0,0 +1,23 @@
+# Local capacity telemetry
+
+The dedicated capacity timer checks every 30 seconds and records a sample only while at least one managed runner is active. This interval captures normal jobs shorter than the five-minute health period without increasing heartbeat or status-report traffic. Samples remain on the controller host at `/var/lib/ci-fleet/capacity/samples.jsonl`; they are never added to status-reporting or heartbeat payloads.
+
+Each sample contains only:
+
+- timestamp and logical controller/pool ID;
+- interval host CPU utilization plus memory, swap, disk-byte, and inode counters;
+- anonymous per-runner CPU percentage and memory use/limit.
+
+Container IDs, names, repositories, jobs, logs, environment variables, source, network counters, and credentials are not stored. The history directory must be a root-owned, non-symlink mode-`0700` directory and the file must be mode `0600`. Malformed records are ignored atomically, records older than eight days are discarded even while the pool is idle, and history is compacted from at most 26,000 records back to 24,000 (more than eight days at the scheduled 30-second interval). Normal samples append one record; compaction rewrites only when retention, malformed input, or the rotation threshold requires it. Uninstall removes this fleet-owned local history. Runner-stat failures fail the capacity service and appear in the normal health report instead of being recorded as zero observations.
+
+## Weekly report
+
+Run on the controller host; reads local state only and changes nothing:
+
+```bash
+sudo /opt/ci-fleet/current/scripts/health.py capacity-report
+```
+
+The JSON report groups the preceding seven days by logical pool and gives nearest-rank p50/p95 values for every observed host and runner metric, plus sample and anonymous runner-observation counts. An empty report means no managed runner was observed during the period.
+
+Review the report before changing runner count, per-runner resources, or pool budget. A capacity change still requires its own reviewed private-configuration change; this public repository does not create or modify that external change.

docs/DESIRED-STATE.md

@@ -88,7 +88,7 @@ The installer:
 10. starts the controller only when its desired state is active and verifies runtime health;
 11. atomically records redacted installation state, then enables the maintenance timers.
 
-A successful second `--install` run reports `NO_CHANGE` and performs no unnecessary replacement. A successful engine upgrade advances both the runtime release and the maintenance installer-manager to the same pinned commit; rollback restores both.
+A successful second `--install` run reports `NO_CHANGE` and performs no unnecessary replacement. Engine upgrades advance the runtime release and maintenance installer-manager together. A deliberate runtime downgrade retains a newer compatible manager so a later reviewed upgrade can still be rendered and activated; maintenance units always come from the selected runtime. Rollback restores both.
 
 ## Adopt an existing controller
 

docs/README.md

@@ -22,7 +22,7 @@ New operator? Use [Operator workflows](OPERATOR-WORKFLOWS.md) to select a suppor
 | Make a project compliant | [Project CI standard](PROJECT-STANDARD.md) and [compliance checklist](COMPLIANCE-CHECKLIST.md) |
 | Split tests across parallel workers | [Project CI standard](PROJECT-STANDARD.md) and the [parallel workflow example](../examples/workflows/parallel-ci.yml.example) |
 | Configure automatic updates and cleanup | [Host maintenance](HOST-MAINTENANCE.md) |
-| Monitor hosts and detect missed reports | [Fleet health monitoring](HEALTH-MONITORING.md) and [authenticated status reporting](STATUS-REPORTING.md) |
+| Monitor hosts and detect missed reports | [Fleet health monitoring](HEALTH-MONITORING.md), [local capacity telemetry](CAPACITY-TELEMETRY.md), and [authenticated status reporting](STATUS-REPORTING.md) |
 | Handle GitHub App, workflow, or deployment secrets | [Secrets model](SECRETS.md) and [security policy](../SECURITY.md) |
 | Review accepted implementation scope | [Design decisions](DESIGN-DECISIONS.md) |
 | Run private CI or deployment for a public project | [Public projects, private delivery, and private configuration](PUBLIC-PRIVATE-CONFIGURATION.md) |

engine-capabilities.json

@@ -1,6 +1,7 @@
 {
   "schema_version": 1,
   "capabilities": {
+    "capacity_telemetry": true,
     "status_reporting_config": true,
     "required_status_reporting": true
   }

host/systemd/ci-fleet-capacity.service

@@ -0,0 +1,21 @@
+[Unit]
+Description=Sample local ci-fleet capacity
+After=docker.service
+Requires=docker.service
+ConditionPathExists=/etc/ci-fleet/ci-fleet.env
+
+[Service]
+Type=oneshot
+ExecStart=/opt/ci-fleet/current/scripts/capacity-sample.sh
+User=root
+Group=root
+NoNewPrivileges=yes
+PrivateTmp=yes
+ProtectHome=yes
+ProtectSystem=strict
+ReadWritePaths=/var/lib/ci-fleet
+LockPersonality=yes
+MemoryDenyWriteExecute=yes
+RestrictAddressFamilies=AF_UNIX
+SystemCallArchitectures=native
+UMask=0077

host/systemd/ci-fleet-capacity.timer

@@ -0,0 +1,11 @@
+[Unit]
+Description=Sample active ci-fleet capacity every 30 seconds
+
+[Timer]
+OnActiveSec=15s
+OnUnitActiveSec=30s
+AccuracySec=1s
+Unit=ci-fleet-capacity.service
+
+[Install]
+WantedBy=timers.target

scripts/capacity-sample.sh

@@ -0,0 +1,16 @@
+#!/usr/bin/env bash
+set -Eeuo pipefail
+
+script_dir=$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)
+environment=/etc/ci-fleet/ci-fleet.env
+args=(capacity-sample)
+if [[ ${CI_FLEET_TESTING:-0} == 1 && -n ${CI_FLEET_ROOT_PREFIX:-} ]]; then
+  environment="$CI_FLEET_ROOT_PREFIX/etc/ci-fleet/ci-fleet.env"
+  args+=(--root "$CI_FLEET_ROOT_PREFIX" --history "$CI_FLEET_ROOT_PREFIX/var/lib/ci-fleet/capacity/samples.jsonl")
+fi
+[[ -r $environment ]] || { printf 'CRITICAL capacity_configuration_missing\n' >&2; exit 2; }
+set -a
+# shellcheck disable=SC1090
+. "$environment"
+set +a
+exec python3 "$script_dir/health.py" "${args[@]}"

scripts/desired_state.py

@@ -30,6 +30,7 @@
 HOST_OPTIONAL = {"CI_FLEET_RUNNER_TTL"}
 REQUIRED_STATUS_CAPABILITY = "required_status_reporting"
 STATUS_REPORTING_CONFIG_CAPABILITY = "status_reporting_config"
+CAPACITY_TELEMETRY_CAPABILITY = "capacity_telemetry"
 
 
 class DesiredStateError(ValueError):
@@ -194,6 +195,8 @@ def build_rendered_env(
         "CI_FLEET_VERSION": short_commit,
         **validate_host_values(host_values),
     }
+    if CAPACITY_TELEMETRY_CAPABILITY in (engine_capabilities or set()):
+        rendered["CI_FLEET_POOL"] = controller["pool"]
     reporting_configured = "status_reporting" in controller
     reporting_required = (controller.get("status_reporting") or {}).get("enabled") is True
     if reporting_required and REQUIRED_STATUS_CAPABILITY not in (engine_capabilities or set()):
@@ -292,6 +295,8 @@ def command_validate_engine_capabilities(args: argparse.Namespace) -> None:
         raise DesiredStateError("selected engine does not support status reporting configuration")
     if args.require_status_reporting and REQUIRED_STATUS_CAPABILITY not in capabilities:
         raise DesiredStateError("selected engine does not advertise required status reporting")
+    if args.require_capacity_telemetry and CAPACITY_TELEMETRY_CAPABILITY not in capabilities:
+        raise DesiredStateError("selected engine does not advertise capacity telemetry")
     print("ENGINE_CAPABILITIES_OK")
 
 
@@ -329,6 +334,7 @@ def parse_args() -> argparse.Namespace:
     capabilities.add_argument("--manifest", type=Path, required=True)
     capabilities.add_argument("--require-status-reporting-config", action="store_true")
     capabilities.add_argument("--require-status-reporting", action="store_true")
+    capabilities.add_argument("--require-capacity-telemetry", action="store_true")
     capabilities.set_defaults(function=command_validate_engine_capabilities)
     return parser.parse_args()