robots.txt & crawl rules

Why it is checked

Checks that a valid robots.txt exists — it is the standard way to tell every crawler (including AI ones) what they may fetch.

How it counts

Scored. It carries 10 of the 100 points of the base score. A check that does not apply to your site leaves the pool, so it never counts against you. A scored check that could not be verified is different: the whole check is then reported as incomplete and no base score is shown, because a score missing part of its evidence would mislead.

How results read

The wording below is exactly what a report shows for each result.

• Passed

  In this check, your site looks good on this point — no action needed.

• Needs attention

  In this check, we found something that weakens this signal. It is not fatal, but fixing it makes your site easier for AI answer engines to use — see the suggested steps below.

• Failed

  In this check, no valid robots.txt was found. Crawlers then fall back to guessing, and you lose the standard place to declare your sitemap and AI policies.

• Not applicable

  This check does not apply to the type of site we detected, so it is not part of your assessment.

• Couldn't verify

  We could not determine this item in this check — the available evidence is not enough to tell why. No conclusion is drawn either way.