# The search results page (/nl/zoeken, /nl/en/search, /nl/de/suche, /nl/fr/recherche, /us/search, /de/suche, /de/en/search, /de/tr/arama, /de/ru/poisk, /de/ar/search) is kept out of the index by the page itself, # which serves . It is not refused here, because a Disallow would # defeat that tag rather than back it up: a Disallow only stops the fetch, a page that is never fetched never # delivers its tag, and a search engine that finds a link to it can still list the bare address with no # description under it. The page was refused here until 2026-09-24, and the tag did nothing for as long as it was. # # /api/ is refused, and only /api/: /api/zoeken is the search dialog's JSON and /api/prijsalert the price-alert # form's POST target. Neither is a page, neither can carry a meta tag, so the fetch is the only thing there is to # refuse, and nothing indexable is behind them: what the dialog shows is on the ranking, brand and budget pages. User-agent: * Allow: / Disallow: /api/ # Assistants may read and cite these pages: the ranking and the daily prices are the point, and a citation sends a # buyer here. Named one by one so the choice is deliberate rather than inherited from the rule above, and named in # groups because a crawler obeys only the group that names it: a group per assistant, which is how this stood until # 2026-09-23, handed each of them the search page and the endpoints the rule above keeps out. # # The crawlers are refused the search page, unlike under the star, by choice: it is a query-shaped address carrying # nothing the ranking does not, and keeping a crawler off it costs no citation. That is crawl control. The rule under # the star is about the index a search engine keeps, where an unfetched page can still be listed by address, and there # the tag on the page has to be read to work. One line per market and language, because a rule is a prefix and # /nl/zoeken covers neither /nl/en/search nor /us/search; test/robots.test.ts derives the list. User-agent: GPTBot User-agent: OAI-SearchBot User-agent: ClaudeBot User-agent: Claude-SearchBot User-agent: PerplexityBot User-agent: Google-Extended User-agent: Applebot-Extended User-agent: CCBot Allow: / Disallow: /nl/zoeken Disallow: /nl/en/search Disallow: /nl/de/suche Disallow: /nl/fr/recherche Disallow: /us/search Disallow: /de/suche Disallow: /de/en/search Disallow: /de/tr/arama Disallow: /de/ru/poisk Disallow: /de/ar/search Disallow: /api/ # Meta's crawler for its assistant reads the pages and none of their filtered or sorted states (#1129, 2026-10-16): it # walked the filter pills, every combination a render of its own, about a thousand uncached pages an hour. Those # states say noindex, follow and carry nothing the plain page does not, so an address with a query is refused here. # The plain pages, which it may read and cite, stay open, and its group repeats the rest of the assistants' rules. # Meta's second crawler, meta-webindexer, is in the group too (#1390): 1,044,780 logged requests from 18 to 25 October, # 15% of all, every one on an address with a query string. User-agent: meta-webindexer User-agent: meta-externalagent Allow: / Disallow: /*? Disallow: /nl/zoeken Disallow: /nl/en/search Disallow: /nl/de/suche Disallow: /nl/fr/recherche Disallow: /us/search Disallow: /de/suche Disallow: /de/en/search Disallow: /de/tr/arama Disallow: /de/ru/poisk Disallow: /de/ar/search Disallow: /api/ # The agents that fetch a page for one person's question, live, and crawl nothing (docs/page-audit-2026-09-25.md, D3). # They may use the search page: kept off it they guessed addresses instead, and a guessed /us/robot-vacuums/roomba-j7 # is a 404. The page still says noindex, so nothing they read there reaches an index. /api/ stays refused, as for all. User-agent: ChatGPT-User User-agent: Claude-User User-agent: Perplexity-User Allow: / Disallow: /api/ Sitemap: https://bestatbest.com/sitemap.xml Sitemap: https://bestatbest.com/us/sitemap.xml Sitemap: https://bestatbest.com/de/sitemap.xml