Crawling website content
This page explains how the crawl pipeline discovers public pages, fetches clean text, and refreshes the content that powers the knowledge base.
What does crawling do?
The crawl pipeline discovers and fetches pages from your site, extracts structured page data, runs the GEO audit, and rebuilds the inline context that powers Kai answers.
It starts from the site's root URL, uses the sitemap when available, follows internal links to depth 3, and respects the plan's per-site page limit: 50 on Free, 500 on Growth, or 5,000 on Scale. Crawl activity shows the live job, current URL, discovered and crawled counts, and up to 500 URL events.
How do I start a crawl?
For your first site:
- On Add your first site, enter Site name and Website URL.
- Select Create site. The site is created and its initial crawl starts automatically.
To add another site from Crawl:
- In Add site, enter Site name and Website URL.
- Select Create site. The new site is selected and its initial crawl starts.
The create button becomes a plan-limit message when the account has 1 Free site, 3 Growth sites, or 15 Scale sites.
To crawl an existing site, select it in Sites and select Crawl content. Growth and Scale can run manual crawls. Free includes the initial crawl and daily content checks, but after the first crawl the disabled button reads Manual sync requires Growth.
What do crawl statuses mean?
The job badge uses pending, running, ready, or failed. Individual URL events use:
| Status | Meaning |
|---|---|
discovered | URL found but not yet fetched. |
fetching | Page is being downloaded. |
fetched | Content successfully retrieved. |
skipped | Page excluded by robots.txt or crawl rules. |
failed | Fetch attempt failed because of a timeout, 4xx response, or 5xx response. |
Healthy jobs finish as ready, with important URLs showing fetched. The completion banner reports how many pages were indexed and confirms that inline context was refreshed. Investigate repeated failed events for pages that should be answer sources.
How do I re-crawl changed content?
All plans receive daily content checks. Growth and Scale also get manual sync from Crawl content and can refresh an existing URL through Fetch & add on the Knowledge base page.
For high-impact changes such as pricing, hours, or policies, Growth and Scale users can run a manual crawl after publishing. Free users must wait for the daily check or upgrade; the product does not currently expose an instant-update webhook.
Removing a site permanently deletes its conversations, widget configuration, and GEO audits. The confirmation action is Yes, remove site.