User-agent: * Disallow: /admin/ User-agent: CCBot Disallow: / User-agent: dotbot Disallow: / User-agent: MJ12bot Disallow: / User-Agent: ziba Disallow: / User-agent: SeznamBot Disallow: / User-Agent: SputnikBot Disallow: / User-agent: BLEXBot Disallow: / User-agent: MegaIndex.ru/2.0 Disallow: / User-agent: spbot Disallow: / User-agent: BUbiNG Disallow: / User-agent: 360Spider Disallow: / User-agent: IstellaBot Disallow:/ User-agent: Qwantify Disallow:/ User-agent: Scrapy/1.4.0 (+http://scrapy.org) Disallow: / User-agent: Scrapy Disallow: /