Architecture

Crawler chain & vision OCR

Every URL-based source (url, sitemap, feed, auto) flows through the same multi-tier crawler chain. Each tier tries a different way to get readable text out of the page. The first tier whose extracted text clears the 200-character threshold wins; failures cascade to the next tier.

The chain (highest priority → last resort)

#	Tier	What it does	When it wins
1	`plain_http`	Plain Guzzle `GET` with a Mozilla user-agent, through `GuardedFetcher`: private IPs and cloud metadata services are refused on every hop, and each connection is pinned to the address that was checked. The transfer stops at 20 MB, a compressed body inflates under the same ceiling, and a web page is read up to its first 5 MB (#725). A response that is a PDF (declared `application/pdf`, or starting with the `%PDF-` bytes), including a page that redirects to one, is turned into HTML from the PDF's text with the same parser file uploads use, up to 20 MB (#642). The browser tiers render a PDF to an empty body, so this tier is the only one that can read it. When the site refuses the browser identity with 403 (a bot shield that blocks browsers coming from servers), the tier asks once more under its own name, `{AppName}Bot/1.0 (+APP_URL)`, e.g. `BlengiBot/1.0 (+https://blengi.com)` (#644). A `429` is never asked again at once under another name (#725): the attempt ends, no other tier fetches the page, the job waits as long as the site's `Retry-After` asks (at least a minute, at most an hour), and the next attempts for that site go out under the crawler's own name from the first request. robots.txt is read the way RFC 9309 describes it (#722): a group that names this crawler applies instead of the `` group, `Allow` lines count, and so do `` and `$` in paths; the longest matching rule wins and `Allow` wins a tie. So a site closed to everyone can let only us in with `User-agent: BlengiBot` + `Allow: /`. A customer never has to allow our server's IP.	Server-rendered pages (Blade apps, classic blogs, marketing sites with SSR or static export).
2	`cloudflare_browser_markdown`	`POST /browser-rendering/markdown`. Cloudflare spins up headless Chrome, waits for JS to settle, and emits clean markdown. We wrap it in a minimal HTML envelope so the extractor and chunker stay unchanged.	JS-heavy SPAs whose final DOM is text-shaped (most React / Vue / Svelte sites).
3	`cloudflare_browser`	`POST /browser-rendering/content`. Same headless Chrome, but returns raw rendered HTML. Our `ReadabilityExtractor` takes over.	Atypical layouts where the markdown extractor's heuristics drop content but the regex extractor catches it (tables that don't map cleanly to GFM, deep nested article structures).
4	`cloudflare_vision`	Two-stage: Full-page screenshot via `/browser-rendering/screenshot` (PNG). Workers AI multimodal call to `@cf/meta/llama-3.2-11b-vision-instruct` with an OCR-only prompt. Returns extracted text. Text is wrapped in `<p>` elements so the chunker treats each paragraph as its own unit.	Canvas-rendered slide decks, all-image landing pages, embedded PDF viewers, or any site whose meaningful content lives only in pixels.

How a tier "wins"

The chain measures extracted text length, not raw HTML length. Each tier's output is passed through ReadabilityExtractor (with the same chrome-stripping fallback used downstream). If the extracted text is at least 200 characters, that tier wins and its HTML is cached for 5 minutes against the source URL. If not, the chain falls forward.

This used to be a raw-length check, which let JS-only Inertia shells "win" with 50KB of empty <div> markup and then silently fail extraction downstream. The extraction-aware check fires the vision tier exactly when the extractor would have failed anyway — no wasted retries, no silent zeros.

Cost & safeguards

CLOUDFLARE_BROWSER_DAILY_LIMIT — shared counter for /content, /markdown, and /screenshot invocations. Default 0 (unlimited). Operators on a fixed Cloudflare bundle should set this to a per-day cap that matches their plan.
CLOUDFLARE_VISION_DAILY_LIMIT — separate counter for Workers AI vision calls (these consume Neurons, billed separately from Browser Rendering). Default 0.
CLOUDFLARE_VISION_MODEL — defaults to @cf/meta/llama-3.2-11b-vision-instruct. Override only if Cloudflare retires the model or you've negotiated a different one.
Both daily-cap guards throw a recoverable exception when the cap is hit, so the chain falls forward to the next tier instead of failing the whole crawl. Vision being capped just means the page that needed OCR doesn't get indexed — every other URL still flows.

Reading robots.txt

RobotsTxtParser fetches a site's robots.txt the way RFC 9309 describes (#725):

It asks under the crawler's own name, {AppName}Bot/1.0 (+APP_URL), not Guzzle's default name, which firewalls often block. Up to five redirects are followed, each one checked, and the first 512 KiB are read (the RFC asks for at least 500 KiB).
2xx: the rules apply.
Any other 4xx, or more than five redirects: there is no robots.txt, so nothing is forbidden.
5xx, 429 or a network error: robots.txt is unreachable, and the RFC says the whole site must then be treated as closed for now. CrawlPageJob does not fetch the page, waits (30, 90, 180 seconds) and tries again; if robots.txt stays unreadable the page keeps its last indexed copy and the Sources page says the site's robots.txt could not be read. Before #725 a server error here counted as "everything allowed".
One fetch per site per hour, shared by every page job through the cache, where it used to be fetched again for every page. An unreachable answer is kept for a minute only.

Finding pages safely

Before a page is crawled, our server talks to the customer's site in three more ways: page discovery (the Find pages button and onboarding read the homepage links, robots.txt Sitemap: lines, sitemaps and a few common paths), sitemap expansion for a sitemap source, and robots.txt. Those requests, and the setup helper's homepage read, all go through App\Support\GuardedFetcher (#724):

each hop is checked against the private-address rules and the connection is pinned to the address that was checked;
redirects are followed by the fetcher, never by Guzzle, and a redirect into our own network ends that request while the rest of discovery carries on;
each file is read up to a limit: robots.txt 512 KiB, homepage links 2 MB, a sitemap 4 MB during discovery and 8 MB per file (32 MB per run) during a crawl.

The Find pages endpoint refuses an internal, private or unresolvable address with a 422 before fetching anything. See Security for the rules themselves.

What happens when every tier fails

The chain throws a RuntimeException whose message lists each tier and why it failed (HTTP code, exception message, or "extraction produced N chars under threshold"). CrawlPageJob handles it without throwing, because an unreachable customer site is not an error in this system and must not write ERROR lines that page the operator (#642):

Hard failures (DNS, connection refused, TLS, any 4xx except 429) end at once; another try would get the same answer.
Everything else (429, 5xx, timeouts, empty renders, an unreadable robots.txt) is retried after 30, 90 and 180 seconds (at least 60 for a 429, and as long as the site's Retry-After asks, up to an hour), then ended. On the sync queue, which runs a job once, the first failure is final.
Ending runs CrawlPageJob::failed(), which stamps the reason on the source's error column when the source has no documents (shown in the admin's Sources dialog through SourceErrorPresenter) and logs a crawl.page_unreachable warning. It does not write a failed_jobs row (#750): a website refusing or staying down is its own answer, not a failure of ours, and those rows used to hold the queue's health check at "warn" for months. A PHP Error raised inside the crawler is a bug and still fails through the queue at once, without retries.

A failed crawl keeps the page's last indexed copy

A rate limit, an outage or a firewall says nothing about the page itself, so the document indexed on the last successful crawl stays and keeps answering; the next refresh tries again. Only two outcomes remove it:

The page is gone: HTTP 404 or 410, or a page that answers 200 with "page not found" wording.
robots.txt now forbids it.

Gate pages (login wall, paywall, JavaScript-only shell, cookie wall, bot challenge) keep the last copy too: their wording overlaps with firewall block pages ("Access denied"). The bot-challenge check knows Cloudflare's 2026 interstitial ("Performing security verification", title "Just a moment..."); before, that page cleared the 200-character threshold and was indexed in place of the real page. Before #642 every failure deleted the page's document, so one night of a customer's host answering 429 to our server could have emptied their whole website source.

The `www` → apex certificate fallback

A very common site misconfiguration is a TLS certificate that covers the apex domain (example.com) but not its www host. Fetching https://www.example.com/… then fails every tier with cURL error 60 (SSL: no alternative certificate subject name matches target host name 'www.example.com') — and because that error is classed as a permanent failure, the page would never re-index. Left unhandled, the assistant answers from stale or missing content (a live case: a pricing page's "3,000 conversations / month" going unindexed).

So before giving up, the chain retries the apex host once: it strips the leading www. (preserving scheme, port, path and query) and re-runs the whole chain. The retry is gated strictly on the cert-mismatch signature — a genuinely broken www-only host (500, DNS failure, 404) is not handed a pointless second fetch — and it is self-terminating, since the apex URL has no www. prefix to fall back from. The recovery is logged as crawler.chain.www_apex_fallback.

Browserless deprecation

The BrowserlessClient tier was removed from the default chain in 2026-06 once the vision OCR tier landed. CF /markdown + /content + vision now cover the same ground free (no third-party billing line). The BROWSERLESS_URL / BROWSERLESS_TOKEN env vars stay readable for backwards compatibility — they no longer wire anything by default. A custom service provider can still bind BrowserlessClient manually if a self-hosted Browserless is preferred.