Build your agent

Knowledge sources

Sources are how an agent learns about your business. This page covers every kind of source, the ingestion pipeline, and what to expect after you click "Add". If different areas of your site deserve different first answers, group sources into Smart Segments — retrieval then prefers the segment matching the page the visitor is on.

The Sources page is the first page of an AI employee's Knowledge tab. Sources that could not be read are listed first, with the reason and a Try again button; the heading counts what was read and what was not. The switch Learn the pages visitors open (see Auto-index visited pages) is at the top of the same page.

Your documents outrank your website

Two kinds of knowledge reach the assistant: curated sources you author and maintain (pasted text, Google Docs and Sheets, Notion pages, uploaded files, SQL), and website pages crawled from your own site. They are not equal. A product page is sales writing that happens to mention facts; a maintained document is a deliberate statement of fact.

So curated sources win a close call in ranking, each source is labelled with its origin in the prompt, and when the two disagree — about a product's properties, a number, a value for one place, or any other fact — the assistant is told to follow the curated one and never to borrow website wording to claim a capability your documents don't state. A figure a source states is quoted as written, never worked out again by the assistant. When a disagreement shows up in a chat, fix the side that is wrong: the assistant can follow the more authoritative source, but it cannot tell you which one is out of date. The ranking lift applies to every AI employee, with or without Smart Segments (before card #681 only agents without segments received it). This is precedence, not exclusion — crawled pages still carry prices, stock and links, and still answer everything your documents don't cover. Tune the strength with RAG_CURATED_BOOST (default 0.10); set it to 0 to weigh every source equally.

One page, one voice. The same page can end up indexed twice — for example once by a sitemap as /meppel/ and once by Auto-indexed from visitors as /meppel. When both copies match a question, only the copy with the best match is used, so a page cannot take two answer slots and outvote the one document that disagrees with it. Your own documents are never treated as a copy, even when a pasted text carries the page's URL as its source.

When your website contradicts itself

Every night your AI employee's pages are checked for places where your website says two different things (card #797), and you see them on Home (Needs your attention) and at the top of the agent's Sources page:

Free shipping — two or more different amounts, for example "Gratis verzending vanaf € 75,-" on most pages and "Vanaf € 50,- betaal je geen verzendkosten" on the homepage. A separate amount for another country (België, Germany…) is a policy, not a contradiction, and is left out.
Stock — a product whose own page says in stock while a category or tag page shows "Niet op voorraad" next to its name, or the other way round.
Opening hours (card #828) — access around the clock ("24/7 toegankelijk") on some pages and a daily window ("De toegangstijden zijn van 06:00 tot 23:00") on others. Only sentences about access or opening hours count: a "24/7 AI employee" or cameras that watch "24 uur per dag" are not opening hours.
Delivery time (card #828) — the same delivery promised in different times ("binnen 1-2 werkdagen" and "binnen 3 werkdagen"). Another country, refunds, replies, returns, installation, your terms, privacy and cookie pages (legal maximums such as "binnen 30 dagen") and anything over two weeks are left out; "2 dagen" and "2 werkdagen" count as the same promise.

Your AI employee already follows each product's own page and your policy pages, and it never takes stock from a category page — but visitors read all of them. Until you fix one of the others, when a question touches it your AI employee gives the more specific value — the daily window, the delivery time on your policy page, the free-shipping amount most of your pages state — and says it may vary, instead of picking one at random. Fix them on your website; once a page no longer says both things, the item disappears after the next night's check.

Your product catalog

Every product page your AI employee reads becomes part of a product catalog (card #799): sizes, materials, colours with their stock, and the price, as fields instead of sentences somewhere on a page. It is built from the product data your shop publishes for search engines (the name, price and stock of Schema.org Product), plus the size list and the composition written on the page — a composition counts only when its percentages add up to 100%, so "5% korting" never becomes a material, and when a page states two different compositions the catalog takes the one it states most often, or none when it can't tell (worth fixing on your site: visitors see both). Card #845 (Jan's brief B6): a list after a label such as Materiaal:, with or without an explanation per part ("38% katoen: zorgt voor zachtheid…"), that adds up to 60–97% is used as the page states it and flagged composition does not add up to 100% on the Products page, rather than left empty. Appliances get two more fields when their pages state them: persons ("4 tot 8 personen", "8+ people", from the product name or a "geschikt voor" sentence) and warranty ("10 jaar garantie"); both reach the answers with the other product facts. Colour versions become one product: STAPP Atlas Marine Sokken and STAPP Atlas Raf Sokken are STAPP Atlas Sokken in Marine (op voorraad) and Raf (uitverkocht). A price is stated only when every colour costs the same. A version with a name of its own stays its own product with its own price, also when that name holds a dash: Bouwkruiwagen staal 85 L — soft wiel (€ 162,50) is not Bouwkruiwagen staal 85 L (€ 152,50) (card #814).

When a visitor names a product, or a page used for the answer does, that product's catalog line rides along with the answer. "What is the Walking made of?" and "Do you have the Thermo Super in black?" are answered from the fields, even when no page matched the question well enough on its own. For what it states the catalog is the authority — it beats category and tag pages — but it is not a complete record: a property it doesn't mention can still be answered from the product page. Before an answer is sent, the product facts in it are checked against the catalog and corrected where they contradict it (see Claim check before sending).

Many product pages list specifications as a table — frame, wheel, tyre, weight. Your AI employee can answer from those rows too: each product's specifications then sit next to its price in the facts it answers from, so it never borrows a neighbouring product's frame (card #815). The rows every version of a product agrees on are used, as long as they fit beside the price. This is switched on per AI employee — ask us; Knowledge › Products then shows each product's specifications. Reading them never changes how your pages are searched.

If you keep a product register (a document listing each product with its sizes and materials), the register wins for every product it lists and the catalog fills in the rest; only the register counts as your complete product list. The catalog is rebuilt ten minutes after a product page changes and every night at 07:20, after the re-crawls. Operators can rebuild it by hand with php artisan knowledge:build-catalog --agent=<id>; add --dry-run to see the products without saving them.

See and correct your products

Knowledge › Products (card #803, shown for shops) lists every product your AI employee answers from, exactly as it uses them: the products of your register marked From your product register, the rest From your product pages. A product is marked Check this when its pages leave the catalog unsure — no material, two different compositions (it then uses the one stated most often), or no sizes — and Needs a check shows only those. A missing material or size list counts only when at least half of your products have one, so a shop that never states materials is not asked about them. While any product needs a check, Home says so under Needs your attention, with a button to the filtered list.

Press Correct it next to a field of a product from your pages and write the right value: the sizes as ranges from low to high (39-42, 43-46), the material, the colours each with their stock, the stock, or one price (12,95 — the currency your pages gave is kept). Your AI employee uses the correction straight away, in its answers and in the claim check, and keeps it when the catalog is rebuilt from your pages: a correction is stored per product and field and laid over every rebuild. Under a corrected field you see what your product pages say; Use the pages again removes your correction. A product from your register is changed in the register itself. Everyone in the workspace can open the page; correcting needs the right to edit the AI employee.

How much it knows

One number, the same everywhere (card #798): the pages and files your AI employee has read. Home says "Knows 111 pages. From 2 sources.", the Knowledge page shows the same 111 next to the number of passages — the pieces it searches when a visitor asks — and each source on the Sources page shows how many of those pages it brought in. A source shows no "of" total: what used to stand there was the crawler's page limit, not the size of your sitemap.

Source types

Type	Use it for	What we ingest
`url`	One specific page — or, with Also crawl the pages this page links to, a hub page and its section	Crawl + extract main content + chunk + embed; with follow-links on, discover same-section links first (depth 1–2) and crawl each
`sitemap`	A whole site at once	Read sitemap, fan out to one `CrawlPageJob` per URL
`feed`	RSS / Atom blogs	Same as sitemap but reads `<item>` entries
`text`	FAQs, snippets, anything you can paste	Skip the crawl, chunk + embed directly
`notion`	Notion pages	OAuth into Notion, fetch via API, treat each page as a document
`google_doc`	Google Docs (Workspace)	OAuth, fetch via Drive API, ingest as a document
`google_sheet`	Google Sheets tabs	OAuth, pull every non-empty row, one Document body with `Row N: …` markers
`sql`	Remote MySQL / PostgreSQL	Direct PDO read-only SELECT; one Document body with `Row N: …` markers. Credentials encrypted at rest with the app key (AES-256-CBC).
`file`	PDF / DOCX / XLSX uploads	Parsed via Cloudflare `toMarkdown` (free tier), chunked + embedded
`woocommerce_products`	WP / WooCommerce stores	Synced by the companion WordPress plugin
`auto`	Pages visitors land on	Auto-queued by `AutoIndexPageVisit` from `/v1/widget/init`

Editing a pasted-text source

Pasted-text (text) sources can be edited in place — no need to delete and recreate. The body is stored on the source's config when you add it, so the Edit (pencil) button on a text row reopens the form pre-filled with the current title, source URL, and content. Saving re-runs IndexTextSourceJob, which replaces the old indexed content: the prior document and its vector chunks are purged before the new text is embedded, so stale answers never linger alongside the edit.

Older text sources created before this feature shipped have no stored body, so their Edit form opens with an empty content box — paste the updated content once and it persists from then on.

Reindex on a pasted-text row re-embeds the saved body — it does not fetch anything external (there is no external origin), but because the body is stored it fully restores a text source that an earlier bug left failed. To change the content, use Edit instead; a text source with no saved body (a legacy one) tells you to edit or re-add it rather than doing nothing.

"What the AI sees" is a preview, not the full text

The grey blurb under each source — and the chunk list in the What the AI sees panel — is a preview, capped for display only. The collapsed blurb shows the first ~600 characters of the first chunk; the expanded panel shows up to 20 chunks, each trimmed to 4,000 characters. This is purely a display limit: the entire source is stored, chunked, and embedded. If a pasted document looks like it "stops" partway (e.g. mid-list), that is the preview boundary, not lost content — the rest lives in the later chunks and is fully answerable.

Hard limits that do apply: a pasted-text source is capped at 200,000 characters at submission — paste more than that and the form rejects it with an error (it is never silently truncated). Chunking adds trailing overlap between adjacent chunks, so facts spanning a chunk boundary survive retrieval. Use the per-chunk character count in the panel to confirm a chunk's full length; a "+N more characters" note appears when a chunk is longer than the on-screen preview.

Spreadsheets: one row, one chunk

A table too big for one chunk — an uploaded Excel or OpenDocument sheet, or a Google Sheet — is indexed one row per chunk. Each row chunk starts with the file's name and the sheet (Waterhardheid Nederland (2) — Blad1) and spells the row out with its column names (postcode_4: 7955; plaatsnaam: IJhorst; waterbedrijf: WMD; hardheid_gem: 8.1; categorie: Gemiddeld); empty cells are left out. A question about one row ("wat is de waterhardheid in IJhorst?") then finds exactly that row. Before card #678 a large table was cut by size into chunks of some 28 rows without column names, and a single place in a 4,000-row sheet was practically never found. Small tables inside a page or document stay with their text. A sheet of more than 5,000 rows puts a few neighbouring rows in each chunk, so one file never becomes more than 5,000 chunks. A sheet indexed before this change needs Reindex once.

Codes and postcodes are looked up exactly

A bare number means almost nothing to the search that finds sources by meaning: a visitor who typed only their postcode, 7955, used to get no answer even though the row postcode_4: 7955; plaatsnaam: IJhorst was in the uploaded sheet. So when a message holds a code — four or more characters with a digit, such as a postcode, an article number or X-200 — that code is also looked up literally in your own documents (uploaded files, sheets, pasted text, SQL, Notion, Google Docs; never in crawled pages). A full Dutch postcode like 7955AB also finds a sheet keyed on 7955. Only the whole code counts (7955 never matches 17955), and a number found in more than three places — a year, 1000 liter — is not treated as a code. A match leads the sources for that answer, whichever segment it is filed under.

Figures are quoted exactly

On every answer (not just Strict mode), the assistant is instructed to reproduce any figure from your sources exactly as written — prices, plan names and quota numbers keep every digit and thousands separator. A small model left to paraphrase will happily turn “3,000 conversations / month” into “3 conversations”, which changes the meaning rather than just the wording. The prompt pins the exact form so the number a visitor reads matches the number on your page.

SQL database source (MySQL + PostgreSQL)

Connect a read-only MySQL or PostgreSQL database directly as a knowledge source. Every non-empty row your SELECT returns becomes part of the agent's training data, with citations the LLM can use as Row N references.

Setup steps:

Open /app/agents/{id}/sources, scroll to Add SQL database.
Pick the driver (MySQL or PostgreSQL) — port autofills to 3306 or 5432.
Paste host, database name, username, password.
Paste a SELECT query — every row that returns becomes part of the agent knowledge base.
Optional column mapping: pick a title_column (used as the row label) and a body_column (used as the row content). When omitted, every column is concatenated as colname: value pairs.
Click Connect & sync. The first SyncSqlSourceJob runs on the crawl queue, indexes the rows, and flips the source to indexed.

Hard rules enforced on every query:

SSRF guard. Host runs through UrlSafetyGuard::assertSafe() — same allowlist the crawler uses. 127.0.0.1, RFC1918 ranges, link-local 169.254.0.0/16 (AWS metadata IP), and any name that resolves to a private IP are all refused. The source flips to failed with "Unsafe SQL host: Refusing to crawl an internal / loopback / link-local host."
SELECT-only. Query must start with SELECT (case-insensitive). Multi-statement (containing ;) is rejected. The keywords INSERT, UPDATE, DELETE, DROP, ALTER, TRUNCATE, GRANT, REVOKE, CREATE, REPLACE, CALL, EXEC, EXECUTE, MERGE, LOAD are blocked.
Read-only transaction. The connector wraps the query in START TRANSACTION READ ONLY (MySQL) / BEGIN TRANSACTION READ ONLY (PostgreSQL) so even a permissive query string can't mutate state.
5,000-row cap per sync. Protects worker memory and the Document table from runaway queries. Lower-bound your SELECT with a LIMIT if you have a big table you don't want fully indexed.
10-second connect timeout. Hosts that can't be reached fail fast — buyers see the error inline.

Credentials at rest:

The host, port, database name, username, and password are stored in sources.credentials_encrypted (text column) and encrypted via Laravel's encrypted:array cast — AES-256-GCM with the install's APP_KEY. The non-sensitive bits (driver, query, title/body column mappings, optional label) live in the regular config JSON column.

Reading the raw column produces ciphertext only; the plaintext is visible only inside worker memory during a sync run and never logged. Rotating APP_KEY will require buyers to re-enter the password (standard Laravel behaviour — see Security).

What gets indexed:

Each row becomes a labeled line in a single Document body. When you set title_column = "title" and body_column = "body", a row with { title: "Welcome", body: "Hello world" } renders as:

Welcome: Hello world

Without a body column it falls back to a key/value join of every non-null column:

Row 1 — id: 42, name: ACME Corp, plan: Pro, last_seen: 2026-05-13

The Document body is then chunked and embedded through the same IndexDocumentJob pipeline every other source uses, so SQL rows surface in retrieval just like crawled pages or pasted text.

Re-sync: with the source's Reindex button, and once a week by pitchbar:refresh-stale-sources (daily at 03:30, for sources last synced more than 7 days ago; it takes the same path as Reindex). (Notion, Google Doc and Google Sheet sources are different: pitchbar:sync-oauth-sources re-syncs them every hour.)

Pages that are not knowledge (card #844): besides cart, checkout, login and account pages, thank-you pages (bedankt, thank-you, danke, merci, order-received), test pages and theme sample pages (test, sample-page, hello-world, staging) are skipped and, on the next crawl, removed. A page that asks search engines not to index it (<meta name="robots" content="noindex">) or that declares another page on the same site its original (rel=canonical) is dropped too, unless it is exactly the address you added as the source. A canonical to the home page is ignored, because some sites point every page there. When your site has a folder per language (/en/, /de/), a question answered in Dutch leaves the pages from other language folders out as long as Dutch pages, or pages without a language folder, were found.

Text repeated across your pages (card #848): a sentence your site repeats on 30% or more of its pages — a standard explainer, "Bel naar … of mail naar …", a review form, size advice — is kept on one page only (the one with the shortest address, usually your home or contact page) and left out of the others when they are indexed, so each page's own content is what answers. Pages indexed before this were cleaned with php artisan knowledge:strip-boilerplate (--agent=, --dry-run).

Every page, not just every source (card #841): a source re-crawl re-reads the pages it finds again, so a page that left the sitemap, sat past the crawl limit or was learned from a visit could keep an old copy for weeks. pitchbar:recheck-stale-pages (daily at 04:45) re-checks every crawled page not checked for 7 days, and product and price pages (prices, tariffs, offers, promotions in the address) after 1 day, at most 400 pages a run, two seconds apart. A page that answers 404 or 410, or redirects to another address, leaves the knowledge with its chunks; a redirect to the same address (https, www, a trailing slash) does not count. Any other page is re-read the normal way, which refreshes a changed copy or stamps it as checked. A site that cannot be reached is tried again the next day.

What the weekly refresh does per source: web pages, sitemaps and feeds are crawled again; pages learned from visitors are re-crawled page by page; SQL sources are re-synced. Pasted text and uploaded files are left alone — nothing outside the dashboard can change them.

Page types (card #780). Every crawled page is marked as a product page (it carries the shop's own product data), a listing (a tag, category, collection or brand overview) or a policy page (shipping, returns, FAQ, terms, privacy, size guide, payment, delivery, warranty); other pages have no type. A listing shows many products in passing, with stock badges a shop does not always keep current, so it loses a near-tie to the product's own page in retrieval, and the assistant is told an overview page never decides one product's stock, price, colours or sizes when that product's own page is among its sources. A listing's stock badges (op voorraad, niet op voorraad, uitverkocht, out of stock…) are removed before the assistant reads it, because shops rarely keep them current (card #790); it takes stock only from a product's own page. At most two listings are used for one answer (card #785): a shop's tag pages often repeat the same size filter and product grid, and without a limit they took every source slot, so no product page reached the answer. Listings beyond the first two give way to the next product or other page that matches well enough, and are only used when nothing else does. The page the visitor is on never counts towards the two. A question about sizes — one that names a size (maat 47, mt 45, size 42) or asks about sizes — also lifts your size guide (a page at an address such as /maattabel, /size-guide or /size-chart) the way a shipping question lifts your shipping page (card #786); your other policy pages are not lifted for it. When the search did not bring the size guide up at all — many tag pages that repeat your size filter can push it out — the part of the guide that best matches the question (the size the visitor named counts most) is added as one extra source. It never raises the answer's confidence, and nothing is added when the search found nothing. A question typed in a hurry — "sokn vor wrk in de kou mt 45" — is repaired from your own pages' words when the search finds nothing (card #787): a word your pages never use becomes the most common word on your pages that it can turn into by adding up to three letters (sokn → sokken, vor → voor, wrk → werk), and the search runs once more. Words with no such match, greetings and short words such as mt stay as typed; a search that found something is never changed. The conversation debugger shows the repaired wording. A sitemap source also skips account, cart, checkout, wishlist (including Dutch wenslijst) and search pages, as visitor-learned pages and followed links already did, and removes such pages it indexed before. Documents indexed before this got their type from php artisan knowledge:classify-pages.

Since card #779 every crawled page also records when it was last checked (documents.checked_at), next to when its content last changed (fetched_at): an unchanged page confirmed this week no longer looks weeks old. An unchanged page whose indexed text went missing (a failed or lost index run) is indexed again on that check. And a sitemap or follow-links refresh also re-checks the pages it indexed earlier that are no longer listed (up to 200 per run): nothing is removed for being left out of a sitemap — a page that now answers 404 or 410 is dropped by the usual gone-page rule, and a page that still exists is simply confirmed.

Drivers not shipped yet: MSSQL and Oracle. Both require PHP extensions (pdo_sqlsrv / oci8) that aren't bundled by default and aren't universal across CodeCanyon hosts. Open a feature request if you need them.

Add a source

Open /app/agents/{id}/sources (Knowledge › Sources). The page has one tab per source type: Web page, Templates, SQL database, Paste text, Google Sheets, Google Docs, Notion and Upload files. Behind the scenes:

Validate. URLs must be http/https; private hosts (10.x, 192.168.x, 127.x, ::1) are blocked to prevent SSRF.
Create the source row with status = pending.
Dispatch a job — CrawlSourceJob for url/sitemap/feed; IngestNotionPageJob/IngestGoogleDocJob for connected sources; IndexTextSourceJob for pasted text.
The job runs on the crawl queue, fetches content, creates Document rows, then dispatches IndexDocumentJob on the index queue.
The status flips from pending → crawling → indexed (or failed with an error message you can read in the UI).

"Indexing didn't finish" — diagnosing failed pages

A page can land in the Knowledge view with 0 chunks and an amber "Indexing didn't finish" badge. The expanded row now shows the actual error from sources.error when present, plus the crawler used and last-fetched timestamp. Most failures map to one of these:

JavaScript-only / SPA pages with no SSR fallback — Cloudflare Browser Rendering executes JS, but if the app fully hydrates client-side and exposes no scrapeable text, the extractor returns under the 200-character floor and the source is marked failed.
Bot challenge / Cloudflare protection — the fetched HTML is the challenge page, not the real content. Detected via detectBlocker() heuristics.
Login wall — site requires auth; we don't run authenticated crawls.
Soft 404 — many sites return a 200-OK "not found" page when a URL is mistyped. looksLike404() rejects these.
Queue worker behind — the document row was created but IndexDocumentJob hasn't run yet. Wait a minute and refresh; if it persists, check Queue Health.
The site's robots.txt keeps crawlers out — the row says so and names the way in (#721): a group for the crawler's own name, User-agent: BlengiBot with Allow: / on blengi.com ({AppName}Bot elsewhere), lets your AI employee read the site while User-agent: * / Disallow: / keeps search engines out. robots.txt is read the RFC 9309 way (#722; see Crawler chain). Then press Reindex.
The site's robots.txt could not be read — the site answered robots.txt with a server error, "too many requests" or not at all. A crawler must then wait (#725), so the page was not read and keeps what it had; the sweep tries again, or press Reindex later.

Click Reindex on a failed row to retry the same pipeline (URL re-crawl + re-index, or file re-parse for uploads). Persistent failures usually mean the URL itself is unscrapable — try a different page on the same site, or upload the content as a file.

Switching embedding models

The Cloudflare Vectorize index is provisioned at the exact dimension of the embedding model that was active when it was first created. Changing CLOUDFLARE_EMBED_MODEL from a 768-dim model (bge-base-en-v1.5) to a 1024-dim model (bge-m3, bge-large-en-v1.5) or back will cause every IndexDocumentJob to crash with:

Cloudflare 40012: invalid vector for id="...", expected 768 dimensions, and got 1024 dimensions

Blengi now detects this BEFORE sending the upsert, surfaces an actionable error, and ships a recovery command. Known model→dim map (auto-applied when VECTOR_DIM env is unset):

Model	Dim
`@cf/baai/bge-small-en-v1.5`	384
`@cf/baai/bge-base-en-v1.5` (default)	768
`@cf/baai/bge-large-en-v1.5`	1024
`@cf/baai/bge-m3`	1024
`text-embedding-3-small`	1536
`text-embedding-3-large`	3072
`text-embedding-ada-002`	1536

Recovery: drop the existing index, recreate at the new dim, and re-dispatch IndexDocumentJob for every document:

php artisan vector:rebuild-index           # interactive — asks before proceeding
php artisan vector:rebuild-index --force   # for automation / CI
php artisan vector:rebuild-index --dim=1024  # override the resolved dim

The command resets every Source to pending, deletes every Chunk row, drops the Vectorize index, recreates it at the target dim, and queues a re-index job per document onto the index queue. File-backed Documents re-index from the persisted text on disk; URL-only Documents need a manual Reindex click (which triggers CrawlPageJob to re-fetch).

A page and the pages it links to

A Single URL source crawls exactly that page — deliberately: a sitemap fan-out once pulled hundreds of pages into a source nobody asked for. But some pages are hubs whose linked pages are the knowledge: Mooiwater's /waterhardheid/ lists ~260 municipality pages, a product family page lists its variants, a help index lists its articles. Tick Also crawl the pages this page links to on a Single URL source (card #583) and the crawler discovers those links first, then crawls each one.

How deep — the pages this page links to (1), or those pages and what they link to (2). Never deeper.
Stay within — the page's own section by default (/waterhardheid/ follows /waterhardheid/meppel/, never /contact/), or the whole website.
Same website only. Cart, checkout, login, account, search and filter/sort URLs are skipped, as are assets. The per-source page limit (CRAWL_MAX_PAGES_PER_SOURCE) caps the total.
Discovery fetches go through the same crawler chain as the crawl itself, so a JS-rendered or bot-protected hub still yields its links.
Every page found belongs to this source, so a source assigned to a segment puts all of them in that segment — the reason to use this rather than a sitemap, which would drag the sitemap's unrelated pages in too.
Reindex re-runs discovery, so a municipality added to the hub later is picked up without touching anything.
Already have the source? The Crawl options action on its row (card #585) switches the option on or off, or changes depth and scope, and re-runs discovery straight away — pages already indexed are kept.

Where a page's title comes from

The page's own <title>, as Readability reads it. Pages fetched through the Cloudflare markdown tier arrive with a front-matter block (title: "…", then a meta: map); since card #586 that block is turned into the title and dropped from the text — before, it leaked into the first chunk and the page was stored as "(no title)". If a page still hands us nothing, the last URL segment is used ("/waterhardheid/maasdonk/" → "Maasdonk"), so no page is ever untitled. Rows stored before the fix are repaired once with php artisan pitchbar:backfill-document-titles (--dry-run to preview). A Reindex also refreshes a page's title even when its content has not changed (card #588) — a title fix never costs a re-embed.

Auto-discovery

On the sources page, the Discover button takes a domain and probes it for crawlable pages without you having to list them. We:

Read robots.txt for sitemap declarations.
Probe a sitemap directly when present.
No sitemap? Read the homepage itself and extract its own same-origin links — this is how sites with no sitemap at all (common for small webshops) still discover their category and info pages, in whatever language and URL scheme they use. Cart, checkout, login/account and filter/sort URLs are filtered out.
Top up with a small set of common paths when results are still thin: /about, /pricing, /features, /products, /faq, /docs, /help, /support, /contact.
Return a checkable list. Tick which to ingest, hit Add selected.

All discovery requests send a regular browser User-Agent — bot-flavoured agents get blocked at the edge by Shopify and Cloudflare-fronted shops, which would make a perfectly healthy site look empty.

Large sites: when discovery finds more than ~30 pages, the picker groups them by URL section with per-section select-all and a filter box, so a webshop with 1,000 product URLs stays navigable. You generally don't need to index every product — pick the category and info pages; individual product pages are understood automatically from the live page when a visitor views them.

"Auto-indexed from visitors"

Every agent has one row of this name. It is not a source you add; it is a bucket that fills by itself: when a visitor opens a page of your site that we have not crawled, that page is queued and added here (gated by your allowed origins, a private-path filter and a rate limit). Until the first such page arrives the row reads Listening, and there is no Reindex button because there is nothing to reindex yet. Once it holds pages it behaves like any other source: Reindex re-crawls the pages it has collected. A page another source already holds is not added again, with or without its trailing slash (see Auto-index). Pages queue on the same crawler as everything else, so a bucket can show Listening for a while after installation while earlier sitemaps are still being read.

Sitemap fan-out

Adding a Source of type sitemap dispatches one CrawlPageJob per URL in the sitemap, staggered by a small per-page delay so Cloudflare Browser Rendering doesn't rate-limit on burst. The discoverer (SitemapDiscoverer) handles three input shapes:

"Queued" means exactly that. The expansion itself — one fetch of the sitemap — runs on the default queue, so a freshly added source flips from Queued to crawling 0/N within seconds. Its pages then wait on the crawl queue, which the single worker drains last and in order: pages of a source added earlier are read first. A second sitemap added right after a first one therefore shows a rising count only once the first one's pages are done. That is a queue position, not a failure; a failure shows as failed with the reason.

Domain root (https://example.com) — probes /sitemap.xml + /sitemap_index.xml.
Direct sitemap URL (https://example.com/sitemap.xml or https://example.com/products/sitemap.xml) — fetched verbatim. Pre-fix the discoverer used to append a second /sitemap.xml here and 404 the request.
Sitemap-index (the <sitemapindex> XML many CMSes — WordPress, Shopify, Webflow — emit by default) — recurses one level into each child sitemap and aggregates page URLs.

Output is deduped (so a URL listed in two child sitemaps gets indexed once) and capped at services.crawl.max_pages_per_source (default 500, override via CRAWL_MAX_PAGES_PER_SOURCE). The cap used to be 25 — a buyer adding a 100-URL sitemap silently lost 75 pages — the new default is generous enough for most marketing / docs sites. Very large catalogues should split the sitemap by section anyway.

Crawler strategies

The crawler tries its tiers in this order and keeps the first result that yields at least 200 characters of extracted text:

Plain HTTP — first: free and fast, fine for server-rendered pages. No JS execution. When a site refuses it (403 / 429) it retries once under the bot identity.
Cloudflare Browser Rendering /markdown — full JS rendering, with Markdown extracted by Cloudflare.
Cloudflare Browser Rendering /content — full JS rendering, then our own extractor.
Workers AI vision — a screenshot read by a vision model, for canvas-drawn or image-only pages.

Tiers 2–4 run only when CLOUDFLARE_ACCOUNT_ID + CLOUDFLARE_API_TOKEN are set and CLOUDFLARE_BROWSER_RENDERING is not false. Browserless is not part of the chain. See Crawler chain.

Once HTML is in hand, ReadabilityExtractor strips nav, footer, ads, etc., leaving the article body. Pages under 200 chars or detected as 404s are dropped.

File upload parsing

Direct file uploads (Sources → Upload files) are parsed locally first, then handed to the same chunk + embed pipeline crawled pages use. The parser is picked by file extension:

Extension	Parser	Network call?
`.pdf`, `.docx`, `.doc`, `.xlsx`, `.xls`, `.odt`, `.ods`	Cloudflare Workers AI `toMarkdown` when CF creds are configured; `Smalot\PdfParser` / `PhpOffice\PhpWord` otherwise	Yes — one multipart POST per file to `/ai/tomarkdown` (free of cost, 0 Neurons)
`.csv`	`League\Csv` — emits one segment per row formatted as `col: value \| col: value`	No
`.md`, `.markdown`, `.txt`	Plain text, split on H1/H2 headings	No

The Cloudflare path is preferred for binary office formats because Smalot and PhpWord are unreliable on real-world documents: Word- exported PDFs that put body text in one big content stream, scanned PDFs with a thin text layer, and DOCX files with nested tables or text frames all tend to extract poorly. Workers AI's toMarkdown returns structured markdown (headings, lists, tables preserved) which feeds the chunker much better.

Pricing: toMarkdown is free for every format above. Only image-to-markdown conversion consumes Workers AI Neurons (we do not send images). When Cloudflare credentials are absent (BYOK OpenAI customers, fresh installs), or when the Cloudflare call fails, the in-process PHP parsers take over so PDF / DOCX / CSV / TXT / MD uploads never silently break.

Spreadsheet uploads (.xlsx, .xls, .ods, .odt) require Cloudflare Workers AI. There is no local fallback. When those formats are uploaded to a workspace without CLOUDFLARE_ACCOUNT_ID + CLOUDFLARE_API_TOKEN configured, the source row is created with status=failed and the error stamp includes an actionable hint: "Spreadsheet / OpenDocument formats need Cloudflare Workers AI." Admins who need Excel ingestion on a BYOK-OpenAI install should export to CSV (the local League\Csv parser handles that format with no external dependency).

Whichever parser ran, the resulting text is persisted under storage/app/private/uploads/{source_id}/segment-N.txt. That's the file the Reindex button reads — you don't need to re-upload the original to re-index.

Chunking and embedding

The extractor's text goes into Chunker, a recursive splitter that prefers semantic boundaries:

Split on markdown headings, then blank lines (paragraphs).
Pack paragraphs greedily up to a target size (~2000 chars / ~500 tokens).
If a paragraph is too big, fall back to sentence boundaries.
Char-window as the absolute last resort.
Add a small overlap between chunks so cross-chunk facts stay linkable.

Chunks are embedded in batches sized by an estimated-token budget (at most 100 chunks and well under the provider's 60,000-token per-request cap; number- and name-heavy text such as a spreadsheet tokenises at roughly two characters per token, so the estimate is deliberately pessimistic). If the provider still rejects a request as too large, the batch is halved and retried on the same provider rather than handed to a fallback embedder, whose vectors would not match the index dimension and upserted into the vector store with metadata: agent_id, document_id, chunk_id, url, workspace_id, source_id, lang.

Crawl retry policy

Each CrawlPageJob attempts up to 3 times with backoff [30s, 90s, 180s]. The retry path is split by failure class:

Rate-limited (HTTP 429, "too many requests" upstream) — releases back to the queue with a fresh 60-second delay without burning a retry slot. Every fan-out page on the same workspace tends to hit the same 429 wave; the shared wait is productive.
Permanent failure (curl DNS resolve / connection refused, HTTP 400 / 401 / 403 / 404 / 410 / 451, malformed URL) — ends at once so the Source row gets the real reason immediately instead of being stranded behind two more retries that will deterministically fail. Since #750 it writes no failed_jobs row: a website's refusal is shown on the source, not counted as a failed job.
Transient (5xx, network blip) — normal retry with backoff.
Per-job timeout — 90s. failOnTimeout=true so a worker SIGTERM still flips the source to failed with a customer-readable error.

Buyer-facing error messages on the Sources list are sanitized via SourceErrorPresenter — raw upstream JSON envelopes (Cloudflare 401 bodies, Browserless stack traces) get rewritten to friendly lines like "We couldn't reach this page" or "The crawl service is busy right now — we will retry automatically." Operators still see the full raw message under Show details.

Reindex and preview

From the sources list, each row has:

Reindex — re-runs the type-appropriate pipeline: URL / sitemap sources re-crawl, OAuth sources (Google Docs/Sheets, Notion) re-fetch via their API, the Auto-indexed from visitors bucket re-crawls each page it has collected, and uploaded files re-chunk + re-embed from the persisted segment text under storage/app/private/uploads/{source_id}/segment-N.txt — no need to re-upload the original. If the persisted file is missing (pre-persistence uploads or a disk wipe) the already-indexed content keeps working and the UI asks for a re-upload only to refresh it; reindexing never fails a source whose content is intact. Pasted-text sources have no external origin to re-fetch, so Reindex re-embeds the text you saved on the source (which also restores one scarred failed by an older bug) — to change the wording itself, use Edit.
Answer on every page (globe icon) — see below.
Preview — shows the extracted documents and a sample of chunks so you can spot bad extraction (e.g. nav bar polluting the text).
Delete — removes the source; its documents and chunks go with it (database cascade). Its vectors are not deleted from the vector store; retrieval skips them because their chunks no longer exist.

Answer on every page (global sources)

Retrieval prefers knowledge from the page the visitor is on (and, with Smart Segments, from that page's segment). That precision is usually what you want — but some facts are global: your address, opening hours, phone number, email. A visitor might ask for them from any page, yet if that content was only crawled onto (say) the Contact page, it can score just below the confidence threshold everywhere else and the assistant answers “I can’t confirm that.”

Toggle the globe button on a source to mark it “Answer on every page.” Its chunks then get the same lift a current-page match gets — from any page or segment — so the fact is reachable everywhere. An Every page badge appears on the source. The lift is relevance-gated and capped: it only applies when the reranker already judged the fact relevant to the question (so “what are your prices?” never drags the address in), and at most two global chunks are force-included per answer, so a large global source can’t crowd out the real answer.

Best for a small, universally-relevant source (a contact card, an opening-hours note). It takes effect instantly — no re-index — and applies to the whole workspace’s retrieval for that agent.

Marking a whole sitemap or crawl as global gives website pages that lift on every question. The lift is larger than the one your own documents get over website pages, so on a close call a global website page then outranks your uploaded file or pasted text.

Automatic retries

You should never have to babysit the Reindex button. Failures heal themselves on two layers:

In-job retries — every indexing job retries up to 3 times with spaced backoff, so a momentary rate limit or network blip resolves within minutes without surfacing at all.
Scheduled sweep — every 15 minutes, pitchbar:retry-sources re-queues sources that failed for transient reasons (rate limits, timeouts, upstream 5xx, an unreachable queue) and rescues sources stranded in crawling/pending by a dead worker. Capped at 3 automatic retries per failure episode (the counter resets after 24 quiet hours), so a site that is down for a day or a week is tried again three times a day until it answers. A 5xx counts as transient however the crawler words it (HTTP 503, Plain HTTP returned 500, status code 502); until card #677 only the first form did, and one night's outage of a customer site left its pages failed for six weeks.

Permanent problems — a page blocked by robots.txt, a 404, a login wall, an expired Google connection, a file that needs re-uploading — are never auto-retried: they stay visible as failed with an actionable message until you fix the cause, and the sweep never burns crawl quota on them.

The knowledge email

Once a day (pitchbar:check-source-health, 08:30) the workspace's owners and admins get one email listing the sources that failed for a permanent reason, once per outage. Since #721 it says which kind of failure each source has. A source that still holds content stopped updating: your AI employee answers from what it last read. A source that holds no content, because it was never read or its page is gone, could not be read: your AI employee cannot answer from it at all. Each line gives the reason in the reader's language, and the visitor bucket keeps its name, Auto-indexed from visitors. Because only failures that are never retried reach this email, it no longer promises automatic retries: it says they will not fix themselves, and Reindex runs the source again once the cause is fixed.

Notion and Google Docs

Both use OAuth. Connect once from /settings/integrations; the token is encrypted at rest. After connecting, the source modal lets you pick pages or documents directly.

Google Docs are exported as Markdown, so paragraph styles marked as Heading 1–6 in the document survive as real headings. The chunker splits at headings and prefixes each chunk with its section heading — which keeps facts from one section (e.g. one product's specifications) from bleeding into answers about another. Structure documents with real heading styles, not bold text: bold lines don't create section boundaries. Docs that can't be exported as Markdown fall back to plain text automatically.

Pasting a docs.google.com document link into the plain URL form also works: the form recognises it and creates a proper Google Doc source (the web crawler can't read Google Docs, so crawling one as a website would always fail). Sheets links get pointed at the Google Sheet tab, which needs the tab name.

Excel files on Drive are not Google Sheets. An .xlsx uploaded to Drive opens in the Sheets editor and its URL still says /spreadsheets/, but the Sheets API refuses to read Office files. The form recognises this (the tell-tale rtpof=true in the link) and explains the fix: open the file, choose File → Save as Google Sheets, and paste the link of the new converted copy — conversion creates a separate file with its own URL.

Notion, Google Doc and Google Sheet sources re-sync every hour (pitchbar:sync-oauth-sources, for sources that are indexed and were last synced more than an hour ago). To pick up a change straight away, click Reindex on that source.

"My agent doesn't know about the file I just uploaded"

Cloudflare Vectorize has eventual consistency on metadata-filtered queries — even after an upsert returns 200 OK, an agent_id-filtered query against that vector typically returns 0 hits for the first 30 to 60 seconds while the metadata index propagates across edge regions.

Practical consequence: a freshly uploaded file shows up as status=indexed in the Sources page immediately, but the agent won't be able to answer questions about it until the propagation window closes. The upload-success banner reminds the admin of this. If the agent still doesn't return relevant chunks after a minute, open the source's Preview to confirm the extracted text isn't empty — that's a parser-side issue, not a vector-side one.

Same gotcha applies to the very first upload after creating a Cloudflare Vectorize index for the first time — the index itself has a ~2 minute provisioning lag before any queries return results, even unfiltered ones.

Storage and retention

Postgres — sources, documents, chunks (text + metadata).
Vector store — embeddings. Cloudflare Vectorize when configured, Qdrant otherwise.
R2 / object storage — original artifacts (PDFs, images) when uploaded.

Deleting a source deletes its row, and the database cascade removes its documents and chunks. Its vector points stay in the vector store (retrieval skips them, because their chunks are gone). There's no soft-delete on sources.