Add initial robots.txt.

Contains disallows for several known AI/GPT related bots.

robots.txt

@@ -0,0 +1,34 @@
+# Used for many other (non-commercial) purposes as well
+User-agent: CCBot
+Disallow: /
+
+# For new training only
+User-agent: GPTBot
+Disallow: /
+
+# Not for training, only for user requests 
+User-agent: ChatGPT-User
+Disallow: /
+
+# Marker for disabling Bard and Vertex AI
+User-agent: Google-Extended
+Disallow: /
+
+# Speech synthesis only?
+User-agent: FacebookBot
+Disallow: /
+
+# Multi-purpose, commercial uses; including LLMs
+User-agent: Omgilibot
+Disallow: /
+
+# Block imagesift.com bot, from thehive.ai
+User-Agent: ImagesiftBot 
+Disallow: /
+
+User-agent: PerplexityBot
+Disallow: /
+User-Agent: anthropic-ai
+Disallow: /
+User-Agent: cohere-ai
+Disallow: /