• Partnership
  • Blog

On Demand Businesses 101

What does the robots.txt file contain?

The robots.txt file for marketplaces allows Google (and other search engines) to index all pages of your marketplace. Some bots might be restricted for performance reasons.

User-agent: *

Disallow: /*auth$

Crawl-Delay: 5

Sitemap: https://marketplacename.com/sitemap.xml.gz

Related Resources

No Comments »

Leave a Reply

Jungleworks