# http://www.robotstxt.org/robotstxt.html User-agent: * ### PORTAL # Disable access to Discovery search and filters Disallow: /aggregator/search/* Disallow: /aggregator/user/* ### ALL PHOTO # NOTE: This could work /lor*simple-search ! tested on google webaster tools Disallow: /*simple-search ### SEALS Disallow: /seals/ ## Config for some chatty bots User-agent: AhrefsBot Crawl-Delay: 60 User-agent: bingbot Crawl-delay: 5