AnnounceHost heartbeats can be starved by SyncHost streams contending on the shared network statistics mutex

[xiemingxun]

Description

Under high download concurrency, dfdaemon may create a large number of short-lived
SyncHost streams. These streams can block AnnounceHost heartbeat generation for
minutes, causing the Scheduler to remove an otherwise healthy host.

The dfdaemon remains connected to the Scheduler and AnnouncePeer requests still
arrive successfully, but they fail with:

rpc error: code = NotFound desc = host <host-id> not found

The proxy request is then eventually returned as HTTP 500.

### Environment

- dragonfly-client: v1.4.4
- Scheduler: v2.5.0
- scheduler.announceInterval: 10s
- Linux bare-metal deployment

The relevant locking and connection lifecycle appear to remain unchanged in
dragonfly-client v1.5.1.

### Root Cause

AnnounceHost and every server-side SyncHost stream use the same cloned
SystemMonitor.network instance.

Both paths call:

network.get_stats().await

Network::get_stats() acquires a shared Arc<Mutex<()>> and holds it while
sleeping for the one-second sampling interval:

let _guard = self.mutex.lock().await;
tokio::time::sleep(Duration::from_secs(1)).await;

Relevant call paths:

- dragonfly-client/src/announcer/mod.rs
    - make_announce_host_request()
    - calls system_monitor.network.get_stats().await before sending the heartbeat

- dragonfly-client/src/grpc/dfdaemon_upload.rs
    - every SyncHost handler repeatedly calls network.get_stats().await

- dragonfly-client-util/src/sysinfo/network.rs
    - serializes all network sampling with one mutex

When many SyncHost tasks are queued, make_announce_host_request() cannot finish,
so the heartbeat RPC is never sent.

Disconnected SyncHost streams can make the backlog worse: a task waiting for the
network mutex cannot observe that its receiver has disconnected until after it
acquires the lock, completes the one-second sample, and attempts to send the result.

### Why So Many SyncHost Streams Are Created

In dragonfly-client/src/resource/parent_selector.rs:

1. Every parent download round calls ParentSelector::register() for all candidate
   parents.

2. A SyncHost stream is created for each parent host not currently registered.
3. A scope guard calls unregister() when the round finishes.
4. When active_requests reaches zero, the connection is immediately shut down and
   removed.

5. There is no idle TTL or grace period for reusing the connection.

Short downloads, range requests, and rescheduling therefore repeatedly create and
destroy the same streams.

### Production Evidence

With a 10-second heartbeat interval, we observed heartbeat gaps of:

31s, 144s, 462s, 1251s, 460s

New SyncHost stream rates were approximately:

97-290 streams/minute on one host
up to 880 streams/minute on another host

During the incident:

- dfdaemon and Scheduler had not restarted.
- AnnouncePeer continued reaching all Schedulers.
- No SyncHost client connection errors occurred during the peak minute.
- Scheduler removed the host after its heartbeat age exceeded twice the announced
  interval.

This rules out a general dfdaemon-to-Scheduler network failure.

### Expected Behavior


- Network statistics sampling should not be performed independently for every
  consumer while holding a shared mutex for one second.

- Short-lived download rounds should reuse recently created SyncHost connections.

### Suggested Fixes

1. Run one background network statistics sampler and let all consumers read cached
   or broadcast values.

2. Decouple AnnounceHost heartbeat generation from the mutex used by SyncHost.
3. Add an idle TTL before removing a parent-host SyncHost connection.
4. Allow a disconnected SyncHost task to be cancelled while waiting for network
   statistics.

5. Add a regression test where many concurrent SyncHost requests cannot delay
   AnnounceHost beyond its configured interval.

### Workarounds

Reducing the Scheduler Cluster candidate_parent_limit from 3 to 1 significantly
reduces stream creation. Setting upload.disableShared: true protects a specific
host but disables its P2P upload capability.

These workarounds reduce the load but do not fix the underlying heartbeat starvation.

The issue is especially visible under workloads involving many small files or
short-lived range requests. If an HTTP slicing layer forwards each slice as an
independent request to dfdaemon, a single large file can also trigger the same
high-frequency SyncHost connection churn.