• @Zak@lemmy.world
      link
      fedilink
      English
      24 months ago

      If I’m reading this comment right, it’s relying on a mistaken understanding of robots.txt. It is not an instruction to the server hosting it not to serve certain robots. It’s actually a request to any robot crawling the site to limit its own behavior. Compliance is 100% voluntary on the part of the robot.

      The ability to deny certain requests from servers that self-report running a version of their software with known vulnerabilities would be useful.