Welcome to What Blocks What @ 4a42
A site that shows the current state of publications robots.txt
Why?
This project was born out of a professional curiosity to see what the current state of robots.txt files are. Which publications are blocking search crawlers, which are blocking AI platforms etc.