MobXGet the crawler policy
Returns a robots.txt disallowing all automated crawling of this origin, including /docs and /openapi.json. Like /health/*, this route is served ahead of the rate limiter: a crawler that receives a 429 here learns nothing about the policy, and many treat an unreadable robots.txt as permission to crawl everything.
Returns a robots.txt disallowing all automated crawling of this origin, including /docs and /openapi.json. Like /health/*, this route is served ahead of the rate limiter: a crawler that receives a 429 here learns nothing about the policy, and many treat an unreadable robots.txt as permission to crawl everything.
Response Body
text/plain
curl -X GET "https://example.com/robots.txt""User-agent: *\nDisallow: /\n"