robots.txt is a public file at the root of the site. Crawlers read it before they fetch pages. It is a gate. It says which paths a given crawler may request, and which paths it should leave alone. It is not a ranking file, and it is not a login.
Allow and disallow
A normal file allows the site and then blocks a few paths you do not want crawled, such as an admin area or an internal search. The lines name a user-agent, then Allow or Disallow, then a path. Disallow: / is the line that asks that crawler not to fetch anything. If it applies to every crawler, the site is closed.
That line is a common accident. A staging site is blocked so it does not get indexed, and the block is copied to production. The site looks finished in a browser, because browsers do not obey robots.txt. Google’s crawler does. Why a finished site can be missing starts with this check for that reason.
What the file does not do
- It does not hide a URL from someone who already has the link.
- It does not replace noindex. Noindex is for a page you will allow to be fetched but do not want indexed.
- It does not make the pages it allows rank.
- It should point at your sitemap with a Sitemap line, so the list can be found.
Blocking a page in robots.txt and also putting it in the sitemap is a contradiction. You asked for discovery and refused the fetch. Pick one. Pages that should stay out of the index, and still be openable, use noindex and stay out of the sitemap. That distinction is worth keeping straight.
If the homepage scan says crawlers are blocked, open robots.txt before you change a single title.
Read the file as a list of instructions, one group at a time. Each group starts with User-agent. The rules under it apply to that crawler, or to every crawler when the agent is a star. A disallow of a single folder is normal. A disallow of slash is the one to question on a live site. If you meant to block only a staging host, that rule should not be on the public host. Keep a Sitemap line at the bottom so the list of real pages can be found once the gate is open.
Search crawlers and AI crawlers
Search crawlers and answer-engine crawlers are scored. A block of the whole site hurts the technical category, which is 20 percent of the letter, and it also hurts the separate AI-search grade. Training crawlers are named in the report and are not penalised if you disallow them. The points are on the methodology page.
The free scan reads robots.txt for the homepage host. If you have just launched, that is the first file to open.
