# Robots.txt # Disable Yahoo! crawler: https://help.yahoo.com/kb/search/SLN22600.html User-agent: Slurp Disallow: / # https://es.semrush.com/bot/ User-agent: SemrushBot Disallow: / # http://www.twenga.com/bot.html #User-agent: TwengaBot-2.0 #Disallow: / # http://mj12bot.com/index.html User-agent: MJ12bot Disallow: / # https://ahrefs.com/robot User-agent: AhrefsBot Disallow: / # https://ahrefs.com/robot User-agent: linkdexbot Disallow: / # https://moz.com/researchtools/ose/dotbot User-agent: dotbot Disallow: / User-agent: Vagabondo Disallow: / User-agent: BaiDuSpider Disallow: / User-agent: Exabot Disallow: / User-agent: Yandex Disallow: / User-agent: BSpider Disallow: / # http://webmeup-crawler.com/ User-agent: BLEXBot Disallow: / User-Agent: ZoominfoBot Disallow: / User-agent: SafeDNSBot Disallow: / User-agent: PetalBot Disallow: / User-agent: Seekport Disallow: / # Other crawlers setup User-agent: * Crawl-delay: 3 # Directories Magento Disallow: /404/ Disallow: /app/ Disallow: /cgi-bin/ Disallow: /downloader/ Disallow: /errors/ Disallow: /includes/ Disallow: /magento/ Disallow: /pkginfo/ Disallow: /report/ Disallow: /scripts/ Disallow: /shell/ Disallow: /stats/ Disallow: /var/ Disallow: /yd/ Disallow: /feed/ Disallow: /import/ Disallow: /magmi/ Disallow: /mt940/ Disallow: /promasy/ Disallow: /am_feed/ # Paths (clean URLs) Magento Disallow: /index.php/ Disallow: /catalog/product_compare/ Disallow: /catalog/category/view/ Disallow: /catalog/product/view/ Disallow: /control/ Disallow: /contacts/ Disallow: /customer/ Disallow: /customize/ Disallow: /newsletter/ Disallow: /poll/ Disallow: /review/ Disallow: /sendfriend/ Disallow: /tag/ Disallow: /wishlist/ Disallow: /landingpages/ Disallow: /checkout/ # Paths (no clean URLs) Magento # Disallow: /*.php$ Disallow: /*?p=*& Disallow: /*?SID= Disallow: /*?limit= Disallow: /*?price= Disallow: /*ajax= Disallow: /*dir= Disallow: /*order= Disallow: /*p=1$ Disallow: *PHPSESSID # Niet relevante pagina's Disallow: /voorwaarden/s Disallow: /disclaimer/ Disallow: /catalog/seo_sitemap/product/ #Sitemaps Sitemap: http://www.venauto.nl/sitemap/nl/sitemapindex.xml