[unsupervised AI] docs: support Sphinx 9

[Himanshuagrawal4]

Warning

This PR was written autonomously by an AI agent and has not been reviewed
by a human yet. Maintainers should ignore it until the human author has reviewed,
understood, and approved everything that the AI agent wrote.

AI-assisted draft: OpenAI Codex assisted with the initial implementation and validation. The contributor manually reviewed and approved the changes before this draft was opened. This PR will remain a draft until all prose and the PR description have been rewritten and finalized by the contributor in accordance with the project policy.

Closes #9189

Summary

• remove the temporary Sphinx 8 upper bound
• replace the removed autosummary get_documenter helper with local registry-based documenter selection
• retain the existing priority behavior and DataDocumenter fallback

Validation

• Ruff check and format checks pass

• the complete documentation tree renders under Sphinx 9.1.0 without the previous import/API failure

• the build still reports existing optional-dependency and ambiguous-reference warnings unrelated to this change

• Tests added / passed

• Passes pixi run lint (focused Ruff checks pass; the full environment was not available locally)

[github-actions]

Unit Test Results

See test report for an extended history of previous test failures. This is useful for diagnosing flaky tests.

    40 files  ±  0      40 suites  ±0   14h 26m 7s ⏱️ - 16m 29s
 4 160 tests  -   1   3 979 ✅ +  2    178 💤  -  3  3 ❌ ±0 
80 959 runs  +483  76 716 ✅ +433  4 240 💤 +50  3 ❌ ±0 

For more details on these failures, see this check.

Results for commit 0ac6b62. ± Comparison against base commit 40fcd99.

This pull request removes 2 and adds 1 tests. Note that renamed tests count towards both.

distributed.cli.tests.test_dask_worker.test_listen_address_ipv6[tcp:..[ ‑ 1]:---nanny]
distributed.cli.tests.test_dask_worker.test_listen_address_ipv6[tcp:..[ ‑ 1]:---no-nanny]

distributed.tests.test_scheduler ‑ test_stop_services

♻️ This comment has been updated with latest results.