From 73f01f42f83e81516cb4a043ef718a00dc850d5d Mon Sep 17 00:00:00 2001 From: Dave Jansen Date: Tue, 26 Mar 2024 10:09:27 +0900 Subject: [PATCH] Add initial robots.txt. Contains disallows for several known AI/GPT related bots. --- robots.txt | 34 ++++++++++++++++++++++++++++++++++ 1 file changed, 34 insertions(+) create mode 100644 robots.txt diff --git a/robots.txt b/robots.txt new file mode 100644 index 0000000..c81b730 --- /dev/null +++ b/robots.txt @@ -0,0 +1,34 @@ +# Used for many other (non-commercial) purposes as well +User-agent: CCBot +Disallow: / + +# For new training only +User-agent: GPTBot +Disallow: / + +# Not for training, only for user requests +User-agent: ChatGPT-User +Disallow: / + +# Marker for disabling Bard and Vertex AI +User-agent: Google-Extended +Disallow: / + +# Speech synthesis only? +User-agent: FacebookBot +Disallow: / + +# Multi-purpose, commercial uses; including LLMs +User-agent: Omgilibot +Disallow: / + +# Block imagesift.com bot, from thehive.ai +User-Agent: ImagesiftBot +Disallow: / + +User-agent: PerplexityBot +Disallow: / +User-Agent: anthropic-ai +Disallow: / +User-Agent: cohere-ai +Disallow: / \ No newline at end of file