Free tool

Robots.txt checker

domain: rust-lang.org
status: NoRobotsTxt
fetch_status: not_found
robots_url: https://rust-lang.org/robots.txt