# Beforthèque — robots.txt # Formations, certifications, métiers, Parcoursup, CPF # Crawlers IA d'entraînement — bloqués. Google-Extended (Gemini) autorisé le 26/10/2026. User-agent: GPTBot Disallow: / User-agent: ClaudeBot Disallow: / User-agent: CCBot Disallow: / User-agent: anthropic-ai Disallow: / User-agent: cohere-ai Disallow: / User-agent: Bytespider Disallow: / User-agent: Amazonbot Disallow: / User-agent: FacebookBot Disallow: / # Moteurs de recherche — autoriser tout sauf les dossiers internes User-agent: * Allow: / Disallow: /backend/ Disallow: /scripts/ Disallow: /data/ Disallow: /prerendered/ Disallow: /config/ Disallow: /frontend/ Disallow: /node_modules/ Sitemap: https://befortheque.com/sitemap.xml # Robots d'audit SEO (14/09/2026) : bienvenus, mais au pas. Ahrefs, Semrush, # Majestic et Moz honorent Crawl-delay ; a 20 s ils passent a 4 000 pages par # jour au plus. Google et Bing ne sont pas concernes. User-agent: AhrefsBot Crawl-delay: 20 Disallow: /backend/ Disallow: /scripts/ Disallow: /data/ Disallow: /prerendered/ Disallow: /config/ Disallow: /frontend/ Disallow: /node_modules/ User-agent: SemrushBot Crawl-delay: 20 Disallow: /backend/ Disallow: /scripts/ Disallow: /data/ Disallow: /prerendered/ Disallow: /config/ Disallow: /frontend/ Disallow: /node_modules/ User-agent: MJ12bot Crawl-delay: 20 Disallow: /backend/ Disallow: /scripts/ Disallow: /data/ Disallow: /prerendered/ Disallow: /config/ Disallow: /frontend/ Disallow: /node_modules/ User-agent: DotBot Crawl-delay: 20 Disallow: /backend/ Disallow: /scripts/ Disallow: /data/ Disallow: /prerendered/ Disallow: /config/ Disallow: /frontend/ Disallow: /node_modules/ # Baidu : moteur chinois, aucun lecteur francophone ici. User-agent: Baiduspider Disallow: /