How a site is ingested
It requests /sitemap.xml first and only follows links if there is none. A site with a good sitemap is ingested completely; a site without one is ingested as far as its internal links reach.
Navigation, footers and boilerplate are stripped so the agent is not repeating your menu at people.
And scoped to the agent you are building.
A short account of what the company is. This is a separate step. See Agents.
A crawl stops after a fixed number of pages. A site larger than that is ingested partially, and the run still reports as complete — so a large site can end up with an agent that confidently knows the first section of the website and nothing about the rest. If your site is big, check the page count against what you expect rather than trusting the status.
Large sites often publish a sitemap index that points at several child sitemaps. If the crawler is pointed at the index and does not descend into the children, it finds almost nothing — which presents as a nearly empty corpus on a site with thousands of pages. Point it at a specific sitemap if the index yields nothing.
Knowledge
Reception agent · 46 pages · recompiled 2 days ago
| Source | Type | Pages | Last ingested | State |
|---|---|---|---|---|
| northwind.example | Website | 46 | 2 days ago | Complete |
| northwind.example/help | Website | 18 | 2 days ago | Complete |
| Price list 2026.pdf | Document | 4 | 9 Mar | Complete |
| docs.northwind.example | Website | 200 | 2 days ago | Capped |
Scoping
Knowledge is scoped to an agent. If you run agents for several brands — or, as an agency, for several clients — each agent must be scoped to its own sources. Getting this wrong produces an agent that answers a question about one client using another client's content, which is the worst possible failure in this module and is not always obvious from a transcript.
- Scope every source to the agent that should have it, at the moment you add it.
- After adding sources for a second brand, ask the first agent a question only the second brand could answer. It should not know.
- Agency workspaces make this explicit — see Agency.
Diagnosing a bad answer
| Symptom | Likely cause | Check |
|---|---|---|
| Vague about what the company does | Empty or stale brief. | Recompile the brief. |
| Knows the homepage, nothing deeper | No sitemap, shallow link crawl. | Publish a sitemap, re-ingest. |
| Knows a fraction of a large site | Page cap. | Compare page count to reality. |
| Almost nothing ingested | Sitemap index not descended. | Point at a child sitemap. |
| Answers with another brand's facts | Scoping. | Check which sources the agent has. |
| Confidently wrong about a price | Stale content, or a PDF that was superseded. | Re-ingest and remove the old document. |
What this does not do
The agent answers from ingested content. It does not fetch a page during a conversation.
Content behind a login is not ingested.
Re-ingestion is triggered, not scheduled. Content changes do not propagate on their own.
Scoping is per source, not per URL pattern.
Questions
Can we upload documents?
Yes — price lists and FAQs are the common ones.
How often should we re-ingest?
Whenever the site changes materially. Nothing does it for you.
Does it use content from our other modules?
No. Knowledge is what you give this module.
Related
Configuring a Phony agent in erp.io: the brief, voice, behaviour, handover, and the prompt-variable failure that makes an agent promise things it cannot do.
ChatsThe Phony chat widget: putting it on a site, the popup behaviour, scoping per property, and the edge cache that hides a working deployment.