For site owners
PlainscrapeBot, and how to control it.
Plainscrape reads public web pages when a customer asks it for data on them. It reads the pages asked for, follows robots.txt, and tells you who it is.
How to recognise it
It says who it is.
| Name in robots.txt | PlainscrapeBot |
| Web Bot Auth key | Ed25519, published at https://www.plainscrape.com/.well-known/http-message-signatures-directory. Signed requests carry Signature-Agent: "https://www.plainscrape.com"; signing is switched on as each bot-verification programme confirms the key. |
| Addresses it sends from | 184.192.177.29, 98.91.85.232, 32.194.198.41 |
robots.txt
Your robots.txt is the rule.
Before reading a page it checks your robots.txt. Rules for PlainscrapeBot apply first; without them, the rules for all bots (*) apply. A page you disallow is not read, and the customer is told why. A Crawl-delay you set is the gap it leaves between pages on your site. To keep it off your whole site:
User-agent: PlainscrapeBot Disallow: /
What it does and doesn't do
It reads what was asked for.
- It reads public pages a customer named, and on a question, a few linked pages on the same site needed to answer it.
- It does not log in, fill in forms or solve CAPTCHAs.
- It does not read sites whose terms forbid automated access.
- Each customer request has a page budget, so it never crawls a whole site on its own.
Contact
Something wrong? Tell us.
Write to hello@plainscrape.com with your domain and what you saw. We answer every message from a site owner.