# robots.txt for www.mmj.com # Medical Marijuana Card Services and Online Telemedicine Evaluation # https://www.mmj.com # # Policy: Allow most legitimate crawlers (search engines, SEO tools, AI systems). # Bytespider is blocked entirely (no reciprocal value, ignores Crawl-delay). # Aggressive scrapers throttled with Crawl-delay. # # Important: robots.txt rules do NOT inherit between User-agent groups. # Any named bot block must repeat the disallows it needs to follow. # That is why the aggressive-bot block below re-lists every Disallow. # =========================================== # DEFAULT RULES (Applies to Google, Bing, AI bots, all unnamed crawlers) # =========================================== User-agent: * # Private/system areas Disallow: /api/ Disallow: /_next/ Disallow: /admin/ Disallow: /private/ # Duplicate content prevention (URL parameters) # Pagination is client-side ("Load More"), so no ?page= disallow is needed. Disallow: /*?*sort= Disallow: /*?*filter= Disallow: /*?*ref= Disallow: /*?*utm_ # Allow static resources explicitly (overrides /_next/ disallow) Allow: /_next/static/ Allow: /images/ Allow: /*.js$ Allow: /*.css$ Allow: /*.png$ Allow: /*.jpg$ Allow: /*.jpeg$ Allow: /*.webp$ Allow: /*.svg$ # LLM-structured content file Allow: /llms.txt # =========================================== # EXPLICIT AI TRAINING ALLOWANCES (Policy signals) # Google-Extended controls Gemini / Bard training use # Applebot-Extended controls Apple Intelligence training use # =========================================== User-agent: Google-Extended Allow: / User-agent: Applebot-Extended Allow: / # =========================================== # BLOCKED SCRAPERS (No reciprocal value) # Bytespider (ByteDance / TikTok) is known to ignore Crawl-delay and # scrapes for internal LLM training without sending traffic back. # =========================================== User-agent: Bytespider Disallow: / # =========================================== # AGGRESSIVE BOTS (Crawl-delay: 3 seconds between requests) # Each block must repeat disallows since robots.txt does not inherit. # =========================================== User-agent: PetalBot User-agent: Brightbot User-agent: Diffbot User-agent: MJ12bot User-agent: BLEXBot User-agent: DataForSeoBot Crawl-delay: 3 Disallow: /api/ Disallow: /_next/ Disallow: /admin/ Disallow: /private/ Disallow: /*?*sort= Disallow: /*?*filter= Disallow: /*?*ref= Disallow: /*?*utm_ Allow: /_next/static/ # =========================================== # SITEMAP # =========================================== Sitemap: https://www.mmj.com/sitemap.xml