fix(httpx): Scope cookies to the session and origin on every request

[Mantisus]

Description

• Session cookies are sent by send_request and stream as well, not only by crawl.
• The Cookie header is rebuilt on every redirect hop, so a cookie collected mid-chain is sent on the next hop, and a cookie that does not match the hop URL is left out.
• Client-level cookies (cookies=, headers={'cookie': ...}) no longer survive a cross-origin redirect. httpx re-applied them from its own jar, where cookies given as a dict get domain='' and match any host.
• Sessions sharing one client no longer see each other's cookies, proxied requests included.
• A Cookie header passed by the caller wins over the session cookies for as long as the chain stays on its origin, matching ImpitHttpClient. An empty value suppresses them.
• persist_cookies_per_session gates storing the response cookies, not sending the session ones.
• Accept, Accept-Language and User-Agent come from a single generated profile instead of two independent ones, so the set is internally consistent.
• HeaderGenerator.get_common_headers is deprecated in favour of get_specific_headers.
• Every proxy now gets a connection pool of its own, so the limits argument applies per proxy.

Testing

• Added new testing to check the processing cookies.

[Copilot]

Pull request overview

This PR updates HttpxHttpClient to unify cookie handling across request paths (including redirects and proxies) and adjusts header generation so the default “browser-like” headers come from a single sampled profile. It also reshuffles/extends unit tests to validate cookie behavior across clients and introduces a deprecation warning for HeaderGenerator.get_common_headers().

Changes:

• Reworked HttpxHttpClient request construction and transport behavior to rebuild Cookie headers per hop and to keep cookie state in Session (including when proxying).
• Updated header generation to request a consistent set of headers from HeaderGenerator.get_specific_headers() (single profile sample) and added tests around this.
• Consolidated cookie/session tests across HTTP clients and added proxy + redirect coverage; deprecated get_common_headers() with a warning and updated tests accordingly.

Reviewed changes

Copilot reviewed 6 out of 6 changed files in this pull request and generated 2 comments.

Show a summary per file

File	Description
tests/unit/http_clients/test_impit.py	Removes Impit-specific cookie/session tests that were moved to shared HTTP client test coverage.
tests/unit/http_clients/test_httpx.py	Adds Httpx-specific tests for same-origin logic, proxy-kwarg behavior, and header sampling behavior.
tests/unit/http_clients/test_http_clients.py	Adds shared cookie/session behavior tests across all HTTP clients, including proxy scenarios.
tests/unit/fingerprint_suite/test_header_generator.py	Updates test to assert the new deprecation warning for get_common_headers().
src/crawlee/http_clients/_httpx.py	Implements unified cookie handling across request paths (redirects + proxies), refactors request building, and optimizes generated header selection.
src/crawlee/fingerprint_suite/_header_generator.py	Marks get_common_headers() as deprecated and emits a DeprecationWarning.

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[vdusek]

LGTM