Skip to content

fix(utils): extract URLs with internationalized top-level domains - #4380

Open
SwastikTripathi wants to merge 1 commit into
apify:masterfrom
SwastikTripathi:fix/extract-urls-idn-tld
Open

SwastikTripathi wants to merge 1 commit into
apify:masterfrom
SwastikTripathi:fix/extract-urls-idn-tld

Conversation

@SwastikTripathi

Copy link
Copy Markdown
Contributor

extractUrls() and downloadListOfUrls() miss URLs on internationalized top-level domains, and cut punycode ones short:

extractUrls({ string: 'https://пример.рф/путь https://xn--e1afmkfd.xn--p1ai/x https://例子.中国/' });
// before: [ 'https://xn--e1afmkfd.xn' ]
// after:  [ 'https://пример.рф/путь', 'https://xn--e1afmkfd.xn--p1ai/x', 'https://例子.中国/' ]

URL_NO_COMMAS_REGEX and URL_WITH_COMMAS_REGEX already accept Unicode letters in the host labels (\p{L}), but the TLD was [a-z]{2,63}. So .рф, .中国 or .みんな never matched, and for a punycode TLD the regex stopped at xn and returned a URL with the wrong host.

The TLD is now xn--[a-z0-9-]{1,59} or [\p{L}\p{M}]{2,63}, with the punycode branch first so xn--p1ai is taken whole. I also added \p{M} next to \p{L} after the first character of the host and in the path, because scripts like Devanagari write vowel signs as combining marks: https://उदाहरण.भारत/page stopped at the first mark before. The first character of the host is unchanged.

The new test (5 URLs, both regexes) fails before the change. test/utils and the enqueue-links tests pass (220), and oxlint/oxfmt are clean.

@github-actions github-actions Bot added the t-tooling Issues with this label are in the ownership of the tooling team. label Nov 11, 2026

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

t-tooling Issues with this label are in the ownership of the tooling team.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant