[jnioche]

Related to #158.

jsoup's Element.selectXpath() is not DOM-free: on every call it converts the whole document to a W3C DOM (NodeUtils.selectXpath → W3CDom.fromJsoup), creates a new XPathFactory and compiles the expression. jsoup.XPathFilter calls it at least once per key and jsoup.LinkParseFilter once per pattern, so a page was converted once per expression evaluated.

Changes

• jsoup.XPathFilter builds the W3C DOM once per page (W3CContext, namespace-unaware as in jsoup's selectXpath) and evaluates all its expressions against it, mapping the results back to the jsoup elements with W3CDom.sourceNodes().
• The XPath expressions are compiled once, when the filter is configured.
• jsoup.LinkParseFilter uses the same shared context.

The extraction syntax (/tidyText(), /allText(), /html(), /@attr) and the results are unchanged.

Performance

Old vs new XPathFilter + LinkParseFilter with the archetype's jsoupfilters.json (6 XPath expressions over 4 keys + 1 link pattern), on the test pages from core/src/test/resources. Average time per page after warm-up, single thread, JDK 17. The outputs (metadata and outlinks) were checked to be identical.

page	size	jsoup parse	W3C conversion	before	after	speedup
stormcrawler.apache.org.html	7 KB	0.05 ms	0.08 ms	0.69 ms	0.38 ms	1.8x
stackexception.html	173 KB	16.7 ms	12.9 ms	75.6 ms	35.2 ms	2.1x
longtext.html	5.7 MB (mostly text)	5.6 ms	0.10 ms	0.69 ms	0.40 ms	1.7x

On a page with a large DOM, a single W3C conversion costs about as much as the jsoup parsing itself; before this change the two filters converted the page ~5 times, taking more than 4x the time spent parsing it. On small pages the gain comes mostly from no longer creating an XPathFactory and compiling the expressions on every call. The gain grows with the number of expressions configured.

Behaviour changes

• An invalid XPath expression now fails when the filter is configured ("Can't compile expression") instead of being logged on every page.
• jsoup's W3CDom.XPathFactoryProperty system property (to plug in another XPath engine) is no longer taken into account; expressions are compiled with the default JDK XPathFactory.

Not covered

Each filter still does its own conversion, so a configuration with both XPathFilter and LinkParseFilter (as in the archetypes) converts the page twice. Sharing the W3C DOM across filters would require caching it per page, which could be stale if a previous JSoupFilter modified the document.

🤖 Generated with Claude Code

[rzo1]

LGTM. Converting once per page and compiling the expressions up front gives the same results as selectXpath and saves a lot. Two small things: LinkParseFilter now returns early when there are no expressions, so it no longer rebuilds the outlinks in that case; that's probably fine, but worth knowing. And LOG.error("... {}: {}", url, e) fills the second placeholder with e, so the stack trace is lost. A small test that an invalid expression now fails in configure() would be nice.

[jnioche]

Thanks @rzo1! Addressed in abb9795: the exception is now passed as a throwable so the stack trace is kept (fixed the two existing "Error evaluating" calls too, and they now include the URL). Added a test that an invalid expression fails in configure(). Kept the early return in LinkParseFilter and added a comment: with no expressions, the old code only rebuilt the outlinks through a HashMap, which removed duplicates and changed their order, so leaving them untouched seems better.

[dpol1]

looks good, will merge when is green, ahm, hopefully