Blog

What does the robots.txt file contain?

jwork 10th April 2020

The robots.txt file for marketplaces allows Google (and other search engines) to index all pages of your marketplace. Some bots might be restricted for performance reasons.

User-agent: *

Disallow: /*auth$

Crawl-Delay: 5

Sitemap: https://marketplacename.com/sitemap.xml.gz

  • Share this article:

You may be interested in these articles

Jungleworks