PYTHON-5805 CSFLE/QE Support for HTTP Proxies

[blink1073]

PYTHON-5805

Changes in this PR

Lets CSFLE and Queryable Encryption tunnel KMS traffic through an HTTP proxy, which networks that force outbound 443 through a proxy currently make impossible. The caller opens the connection and the driver still performs the KMS TLS handshake, so verification targets the KMS host rather than the proxy.

• Adds kms_connect_callback on AutoEncryptionOpts, ClientEncryption and AsyncClientEncryption, plus KMSConnectContext.
• Adds HTTPProxyKMSConnect and AsyncHTTPProxyKMSConnect, so most users pass a helper instead of writing a callback.
• Splits TLS wrapping out of the connect helpers in pymongo/pool_shared.py, leaving existing connection paths unchanged.
• Rejects a callback that returns the wrong thing without retrying. The cause is a ConfigurationError, which public operations surface as an EncryptionError wrapping it. Network errors stay retryable.
• Adds the six prose tests from section 28 of the client-side-encryption test specification.
• Skips prose case 5 as a known specification discrepancy, tracked in PYTHON-6037.

Mostly tests: 1411 of the 1878 added lines, and 797 of the total is the synchro-generated mirror.

Test Plan

Prose tests against real AWS KMS through the drivers-evergreen-tools proxies, both flavors: 10 passed, 2 skipped, 0 failed. The proxy logged connect_target kms.us-east-1.amazonaws.com:443. Unit tests: 41 passed, 1 skipped. just lint, just typing and just docs clean.

Evergreen patch, 7 encryption variants across RHEL8, macOS and Windows: https://spruce.corp.mongodb.com/version/6a873cdeb7e195000786c854

Checklist

Checklist for Author

• Did you update the changelog (if necessary)?
• Is there test coverage?
• Is any followup work tracked in a JIRA ticket? If so, add link(s).
  • PYTHON-6037 Support timeoutMS on ClientEncryption

Checklist for Reviewer

• Does the title of the PR reference a JIRA Ticket?
• Do you fully understand the implementation? (Would you be comfortable explaining how this code works to someone else?)
• Is all relevant documentation (README or docstring) updated?

[codecov]

Codecov Report

❌ Patch coverage is 92.22798% with 15 lines in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
pymongo/encryption_options.py	88.78%	12 Missing ⚠️
pymongo/synchronous/encryption.py	92.30%	2 Missing and 1 partial ⚠️

📢 Thoughts on this report? Let us know!

[Copilot]

Pull request overview

Adds HTTP proxy tunneling for CSFLE and Queryable Encryption KMS traffic while preserving end-to-end KMS TLS verification.

Changes:

• Adds synchronous/asynchronous KMS connection callbacks and HTTP proxy helpers.
• Separates socket connection from TLS wrapping.
• Adds unit, integration, generated mirror tests, and documentation.

Reviewed changes

Copilot reviewed 10 out of 10 changed files in this pull request and generated 7 comments.

Show a summary per file

File	Description
pymongo/encryption_options.py	Defines callback APIs, context, and proxy helpers.
pymongo/asynchronous/encryption.py	Integrates callbacks into asynchronous KMS requests.
pymongo/synchronous/encryption.py	Provides the generated synchronous integration.
pymongo/pool_shared.py	Extracts TLS wrapping from connection creation.
test/asynchronous/test_encryption.py	Tests asynchronous callbacks and proxy tunneling.
test/test_encryption.py	Provides synchronous encryption tests.
test/asynchronous/test_pooling.py	Tests asynchronous TLS wrapping.
test/test_pooling.py	Provides synchronous TLS-wrapping tests.
tools/synchro.py	Adds mappings for new asynchronous symbols.
doc/changelog.rst	Documents HTTP proxy support.

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[Copilot]

Pull request overview

Copilot reviewed 10 out of 10 changed files in this pull request and generated 4 comments.

Suppressed comments (3)

Previously missed (3) — in code that hasn't changed since the last review.

pymongo/encryption_options.py:198

• Thread startup is not failure-atomic. If socket.socketpair() or either Thread.start() fails (for example under descriptor or thread exhaustion), _bridge raises after the caller's close guard and can leave the TLS proxy socket, socketpair, and possibly the first relay thread alive. Add cleanup for every partially created resource before propagating startup failures.

        for pair in ((relay_side, proxy), (proxy, relay_side)):
            threading.Thread(target=relay, args=pair, daemon=True).start()

test/asynchronous/test_encryption.py:2356

• This async helper performs synchronous http.client connect, request, and response reads on the event-loop thread, with no connection timeout. A slow or unavailable proxy can block the entire async test loop indefinitely. Run the complete blocking transaction in a worker thread (or use async networking) while preserving the generated synchronous variant.

            conn = http.client.HTTPSConnection(
                f"{KMS_PROXY_HOST}:{KMS_TLS_PROXY_PORT}", context=ctx
            )
        else:
            conn = http.client.HTTPConnection(f"{KMS_PROXY_HOST}:{KMS_PROXY_PORT}")
        try:
            conn.request(method, path)
            return conn.getresponse().read().decode()

test/asynchronous/test_encryption.py:333

• This coroutine callback calls blocking socket.create_connection directly, which can stall the event loop during DNS resolution or connection setup. Use the event loop's nonblocking socket connection API or offload the connect before returning the deliberately nonblocking socket.

        async def callback(context):
            sock = socket.create_connection(listener.getsockname(), timeout=10)
            sock.setblocking(False)
            return sock

[Copilot]

Pull request overview

Copilot reviewed 10 out of 10 changed files in this pull request and generated 1 comment.

Suppressed comments (2)

Previously missed (2) — in code that hasn't changed since the last review.

pymongo/encryption_options.py:244

• A task cancellation here cancels the asyncio Future but cannot stop an already-running executor thread. If super().__call__ subsequently succeeds, its connected tunnel socket is discarded without being closed; for HTTPS proxies this can also leave both relay threads alive. Retain/shield the worker future and arrange to close its eventual socket result when the awaiting task is cancelled.

        return await asyncio.get_running_loop().run_in_executor(None, connect)

pymongo/encryption_options.py:228

• If socket.socketpair() fails before _bridge reaches its cleanup block (for example under file-descriptor exhaustion), this already-connected TLS proxy socket is leaked. Ensure every _bridge failure closes sock.

        return self._bridge(sock)