VIPnytt / UserAgentParser Star 1 Code Issues Pull requests User-Agent parser for robots.txt, X-Robots-tag and Robots-meta-tag rule sets crawler robot user-agent spider user-agent-parser robots-txt rep robots-tag robots-exclusion-protocol x-robots-tag robots-meta-tag Updated Apr 4, 2021 PHP
RobotstxtGenerator / Robots.txt-Generator Star 0 Code Issues Pull requests The Robots.txt Generator tool helps you to create the Robots.txt file for your website. robots-txt robots-generator robots robots-tag robots-txt-generator robots-txt-creator robots-txt-file Updated Aug 30, 2021
toimik / RobotsProtocol Star 0 Code Issues Pull requests Discussions Parsers for robots.txt (aka Robots Exclusion Standard / Robots Exclusion Protocol), Robots Meta Tag, and X-Robots-Tag robots-txt robots meta-tags robots-parser robots-exclusion-standard robotstxt robots-tag robots-exclusion-protocol x-robots-tag Updated May 2, 2024 C#