pastebin

Paste Search Dynamic
Recent pastes
9gag.com/robots.txt
  1. # Robots.txt file for https://9gag.com
  2. # All robots will spider the domain
  3.  
  4. user-agent: Mediapartners-Google
  5.  
  6. disallow: /read/
  7. disallow: /gag-in-app/*
  8. disallow: /views?*
  9. disallow: /search*
  10. disallow: /search?query=*
  11. disallow: /pref/*
  12. disallow: /settings*
  13. disallow: /notifications*
  14. disallow: /views*
  15. disallow: /connect/*
  16. disallow: /others/*
  17. disallow: /static/*
  18. disallow: /static/ads/*
  19. disallow: /l.php
  20.  
  21. Crawl-delay: 1
  22.  
Parsed in 0.002 seconds