Free tool

Robots.txt checker

domain: rust-lang.net
status: NoRobotsTxt
fetch_status: not_found
robots_url: https://rust-lang.net/robots.txt