crawlable

How many links does your page serve before JavaScript runs?

A bash script of under 100 lines (curl, grep and sed, nothing else) that counts the <a href> elements in the HTML your server actually sends, and how many of them point at other pages of the same site.

Compare that number with what the browser sees, in the console of the same page:

document.querySelectorAll('a[href]').length

The gap between the two is the part of your internal link graph that only exists after render.

Why

Googlebot does render JavaScript, but rendering sits in a separate, budgeted queue. Plenty of other readers do not render at all: link checkers, link-preview generators, archive bots, many smaller crawlers and a lot of LLM fetchers. They read the bytes and leave.

A site whose navigation is drawn by a framework (a three.js scene, a canvas, a client-side router) can look like a complete site map in a browser and like a dead end to a plain fetch, with nothing broken and no audit tool complaining.

Usage

chmod +x crawlable.sh
./crawlable.sh https://example.com/page                      # one page
./crawlable.sh https://example.com/page --list               # also print the hrefs
./crawlable.sh --sitemap https://example.com/sitemap.xml     # every URL in a sitemap

Each line of output gives the URL, the bytes served, the number of anchors and the number of internal links.

Reading the numbers

• anchors: every href in the HTML the server actually sent
• internal: the subset that points at another page of the same site, which is what carries authority around your own site

A page whose whole navigation is drawn by a framework typically returns a handful of anchors and one or two internal links, while an ordinary page on the same site returns dozens. Neither is a bug. The fix is usually not to throw the design away: it is to put a plain list of HTML links underneath it, so the crawler is handed the same map the visitor gets.

If you hide that list visually, test it with the keyboard as well. Links inside a visually-hidden block are still Tab stops, and a keyboard user will walk through them without seeing where focus went.

Background

Written while auditing AI Tools Primer, an independent site of hands-on guides to AI tools and everyday software whose navigation is a three.js island. The first run of this check found that homepage serving two anchors to a crawler.