the crawler

HumanForwardBot

The findability score reads a site the way an AI model would, then tells the owner what the models can and cannot see. This is the crawler that does the reading.

What it does

It fetches the homepage, robots.txt, sitemap.xml, llms.txt, one product page, and a few well known machine endpoints (products.json, /.well-known/ucp, /api/mcp). About eight requests in all. It runs once per domain per fourteen days at most, only when someone asks for a score, and it never submits forms, logs in or crawls the whole site.

How it identifies itself

Allow it

Most sites need nothing. If yours sits behind Cloudflare, Akamai, Imperva, DataDome or a similar bot wall, add a rule that allows the user agent above. In robots.txt:

If you block it, we respect that and show no score. The models you are trying to be found by read sites the same way, so a wall that stops us usually stops them too.

Block it

Questions: hello@humanforward.io