# ========================= # KI / LLM / Data Crawlers # ========================= User-agent: GPTBot Disallow: / User-agent: ChatGPT-User Disallow: / User-agent: OpenAI-Searchbot Disallow: / User-agent: ClaudeBot Disallow: / User-agent: Claude-User Disallow: / User-agent: Claude-Web Disallow: / User-agent: anthropic-ai Disallow: / User-agent: CCBot Disallow: / User-agent: Cohere-Agent Disallow: / User-agent: cohere-ai Disallow: / User-agent: PerplexityBot Disallow: / User-agent: Perplexity-User Disallow: / User-agent: Amazonbot Disallow: / User-agent: Amazon-Adventures Disallow: / User-agent: AmazonProductDiscoverybot Disallow: / User-agent: Google-Extended Disallow: / User-agent: GoogleOther Disallow: / User-agent: Bytespider Disallow: / User-agent: DuckAssistBot Disallow: / User-agent: OAI-SearchBot Disallow: / User-agent: Meta-ExternalAgent Disallow: / User-agent: Meta-ExternalFetcher Disallow: / User-agent: Applebot-Extended Disallow: / # ========================= # SEO / Scraper / Auditor # ========================= User-agent: AhrefsBot Disallow: / User-agent: AhrefsSiteAudit Disallow: / User-agent: SemrushBot Disallow: / User-agent: SEMrushBot Disallow: / User-agent: MJ12bot Disallow: / User-agent: Sistrix Crawler Disallow: / User-agent: XoviBot Disallow: / User-agent: Screaming Frog SEO Spider Disallow: / User-agent: Sitebulb Disallow: / User-agent: Siteimprove Disallow: / User-agent: Sitechecker Disallow: / User-agent: BrokenLinkCheck Disallow: / User-agent: Dead Link Checker Disallow: / User-agent: LinkChecker Disallow: / User-agent: Wget Disallow: / User-agent: HTTrack Disallow: / User-agent: curl Disallow: / User-agent: Python-urllib Disallow: / User-agent: Python Requests Disallow: / User-agent: scrapy Disallow: / User-agent: Scrapy Disallow: / User-agent: Colly Disallow: / User-agent: Playwright Disallow: / User-agent: Headless Chrome Disallow: / # ========================= # Security / Scanner # ========================= User-agent: Burp Disallow: / User-agent: Burp Suite Disallow: / User-agent: Nmap Disallow: / User-agent: sqlmap Disallow: / User-agent: Nessus Disallow: / User-agent: Qualys Disallow: / User-agent: ZGrab Disallow: / User-agent: Wfuzz Disallow: / User-agent: Nuclei Disallow: / User-agent: Vulnerability Disallow: / # ========================= # Monitoring / Uptime / Preview # ========================= User-agent: UptimeRobot Allow: / User-agent: Pingdom Allow: / User-agent: StatusCake Allow: / User-agent: Better Uptime Allow: / User-agent: Slackbot Allow: / User-agent: Discordbot Allow: / User-agent: facebookexternalhit Allow: / User-agent: Twitterbot Allow: / User-agent: Googlebot Allow: / User-agent: Bingbot Allow: / # ========================= # Standard # ========================= User-agent: * Allow: / Disallow: /admin.php Sitemap: https://www.poeten.de/sitemap.php