# Training and bulk-dataset crawlers are blocked. These fetch the site to
# build a corpus, not to send a reader here, so they take the method and
# return nothing. Common Crawl (CCBot) is the one that matters most: its
# archive is public and is a common base for training sets.
#
# Deliberately still allowed in the "*" group below: ordinary search engines,
# the user-directed AI fetchers that only run because a person asked a
# question (ChatGPT-User, OAI-SearchBot, Claude-User, Claude-SearchBot,
# PerplexityBot, Perplexity-User), and the link-preview bots that render
# share cards. Those can send a buyer. Blocking them would cost sales.
#
# Note the pairs that look alike and are not: Applebot stays allowed for
# Siri and Spotlight search, Applebot-Extended is the training opt-out and is
# blocked. ClaudeBot is the training crawler and is blocked, Claude-User and
# Claude-SearchBot are user-directed and are not.
#
# This is containment, not security: it only binds crawlers that choose to
# honour it.
User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: Google-Extended
Disallow: /

User-agent: CCBot
Disallow: /

User-agent: Bytespider
Disallow: /

User-agent: Applebot-Extended
Disallow: /

User-agent: meta-externalagent
Disallow: /

User-agent: anthropic-ai
Disallow: /

User-agent: Claude-Web
Disallow: /

User-agent: cohere-ai
Disallow: /

User-agent: cohere-training-data-crawler
Disallow: /

User-agent: Diffbot
Disallow: /

User-agent: omgili
Disallow: /

User-agent: omgilibot
Disallow: /

User-agent: Webzio-Extended
Disallow: /

User-agent: ImagesiftBot
Disallow: /

User-agent: Timpibot
Disallow: /

User-agent: PanguBot
Disallow: /

User-agent: AI2Bot
Disallow: /

User-agent: AI2Bot-Dolma
Disallow: /

User-agent: FriendlyCrawler
Disallow: /

User-agent: img2dataset
Disallow: /

User-agent: Scrapy
Disallow: /

User-agent: *
Allow: /

Disallow: /admin
Disallow: /api/
Disallow: /dashboard
Disallow: /access
Disallow: /checkout
Disallow: /my-referral
Disallow: /ideas/
Disallow: /report/
Disallow: /embed/

Sitemap: https://demanddiscovery.ai/sitemap.xml