# www.robotstxt.org/ # www.google.com/support/webmasters/bin/answer.py?hl=en&answer=156449 User-agent: * Disallow: User-agent: * Disallow: /pattern-subscribe/ Disallow: /my-account/ Disallow: /checkout/ Disallow: /reviews/pattern/ User-agent: GeedoBot User-agent: AhrefsBot User-agent: SemrushBot User-agent: MJ12bot User-agent: DotBot User-agent: ImagesiftBot User-agent: PetalBot User-agent: YandexBot User-agent: Bytespider Disallow: /