Remove tldextract from core dependencies to drop the requests stack

[vdusek]

Problem

tldextract is a core dependency used in one place — _utils/urls.py (_domain_under_public_suffix()), only for the opt-in same-domain enqueue strategy. The default strategy everywhere is same-hostname, so for most users it never runs. Two issues:

1. It's the only reason the requests stack is in every install. Nothing in src/crawlee imports requests directly. Removing tldextract drops 7 packages / ~1.8 MB:

   tldextract, requests, requests-file, filelock, urllib3, certifi, charset-normalizer
   (idna stays — needed by yarl)
   

2. It hits the network on first use. TLDExtract(cache_dir=tempfile.mkdtemp()) + tldextract defaults → the first same-domain lookup tries to fetch the PSL over the network before falling back to the bundled snapshot (and writes a disk cache, which breaks on read-only filesystems).

What we actually need

Given a host, get its registrable domain (eTLD+1) from the Public Suffix List. For same-domain we only compare two hosts for equality.

Options (with runtime dep counts)

#	Approach	New deps	Effect
A	Vendor a PSL snapshot + ~40-line matcher (like Crawlee JS / tldts)	0	removes all 7 pkgs, no network; we refresh snapshot via CI
B1	Swap for publicsuffixlist	1 (0 transitive)	net −6 pkgs; pure Python, bundles PSL, auto-updated daily, actively maintained
B2	Swap for tld	1 (0 transitive)	net −6 pkgs; pure Python, bundles PSL, slower cadence
B3	Swap for publicsuffix2	1 (0 transitive)	net −6 pkgs; unmaintained since 2019 — avoid
C	Lazy-import + move tldextract to an extra	0 default / 7 for same-domain	keeps network fetch, only hides the problem
D	Naive "last two labels", no PSL	0	removes all 7 pkgs but breaks .co.uk, github.io, … — not acceptable

Recommendation: A (vendored snapshot) — zero deps, no network, full control. If we'd rather not maintain a snapshot, B1 (publicsuffixlist) is the best fallback.

same-domain behavior must stay correct for multi-part suffixes; add test cases in tests/unit/_utils/test_urls.py.

[vdusek]

I believe we should try to do option A, with a regular CI refresh.

[janbuchar]

I'd argue that hitting the network in a web scraping library is a very negligible drawback. Tying updates to the list of existing TLDs to crawlee's development cycle might cause problems for users who can't update for whatever reason. But if we provide some kind of an escape hatch, approach A is fine.

In the end, a custom request transform function for enqueue links gives you all the control you need (unless I'm missing something), so perhaps we don't need additional parameters after all.

[vdusek]

Okay, so my opinion in this case is that we should rather implement the fetching ourselves, in the Crawlee, using our default HTTP client, rather than depend on a library that pulls seven additional packages only to support the same-domain enqueue-links strategy.

[vdusek]

Or maybe use the tld package, which looks okay at first glance and has zero additional dependencies.