Free tool

Robots.txt checker

domain: rust-lang.com
status: NoRobotsTxt
fetch_status: not_found
robots_url: https://rust-lang.com/robots.txt