Clear it in 3 steps
Start the environment
Investigate the target
Submit the flag
All challengesIn progress
Web-04. The robots.txt Trap
Mission
Websites often serve a robots.txt file for crawlers. In this challenge, check whether a file that robots.txt tells crawlers not to index can still be accessed directly.
Not started
Hints
Hint 1Show
Start by visiting /robots.txt
Hint 2Show
Check which path is listed under Disallow
Hint 3Show
Try accessing that path directly in your browser