Stradale
MarketDataBot
We keep a record of collector vehicles by VIN. Our crawler reads public listing pages so a car’s history follows the car instead of disappearing when a listing comes down. We link back to the source on every listing we hold.
It says it is a bot
Every request carries Mozilla/5.0 (compatible; MarketDataBot/1.0). It declares itself a crawler in the clear. We never send a plain browser user-agent, and we never run a headless browser to look like a person.
It obeys robots.txt
We read your /robots.txt before the first request and re-read it twice a day. Rules under MarketDataBot take precedence over *. Longest-match wins, so a narrow Allow inside a broad Disallowworks as you’d expect.
It waits as long as you ask
Crawl-delay is honored to the second, up to 30 seconds. Without one we wait 2 seconds between requests to the same host, and we never open parallel connections to you.
It stops when you say stop
A challenge page, a 403, or a 401 ends the run. We don’t retry through it, rotate addresses, or vary the user-agent. If we can’t reach you honestly, we’d rather ask for a feed.
Blocking us
Add this to your robots.txt and we’ll stop:
User-agent: MarketDataBot Disallow: /
It takes effect within twelve hours. To have your data removed as well, write to ingest@stradale.xyz and we’ll delete it.
Working with us instead
If you’d rather send us a feed than be crawled, we’d prefer that too: it is faster for you, cleaner for us, and you keep control of what we hold. Write to partners@stradale.xyz.
See also our privacy policy and terms.