Robots.txt is a text file belonging to a group of web protocols called Robots Exclusion Protocol or REP. This file commands search engine bots like Googlebots on what pages to crawl and index. Robot.txt helps websites avoid being bombarded by too many requests. It plays a huge role in having a good SEO score.

Updated January 2, 2023

Looking for SEO Software?

Compare Software

Want to suggest a term?

Know something we’re missing in this dictionary? Let us know! You can reach out via our contact form.

Let us guide you so that you can make the right choice

Simplify your software search and get started by browsing our categories.

Browse our categories

© 2023, a SoHo Media Ltd company. All Rights reserved.
Advertising Disclosure | Cookie Policy | Privacy Policy | Terms of Use | Sitemap