6 questions before you build a competitor scraper

Somebody on your team has already said it. "We could just write a script for that." They're right. A headless browser, a list of URLs, a cron job, and a Slack webhook will get you a competitor monitor by Sunday night.

The questions below aren't meant to talk you out of it. They're the ones that decide whether the script is still useful in March or quietly dead by November.

1. What exactly counts as a change?

This is the question that sinks most homegrown monitors, and it comes up in the first week. A raw HTML diff fires on everything: rotating testimonials, cookie banners, A/B test variants, a CSRF token in a hidden field, a "last updated 3 minutes ago" timestamp in the footer.

So you start stripping things. First scripts, then a few CSS selectors, then a regex for dates. Each rule is small. Together they turn into a filter nobody else on the team understands. We wrote up the most common offenders in 6 ways a competitor diff lies to you. Before writing any code, decide whether you're tracking pixels, rendered text, or specific facts like prices and plan names. Those are three different systems.

2. Who fixes it when a competitor redesigns?

Competitors redesign without telling you. When they do, selector-based extraction doesn't fail loudly. It returns an empty string, the diff reads "price removed," and someone in sales gets a very exciting and completely wrong alert.

Name the owner now. If the answer is "whoever built it," check whether that person will still be on the team, and still interested, in nine months.

3. How will you know it stopped working?

This one is sneakier than breakage. A scraper that throws errors is easy to spot. A scraper that runs fine against a bot-challenge page, a login wall, or a regional redirect produces a clean, stable, useless snapshot every day. No diffs, no alerts, and it looks exactly like a quiet market.

You need a heartbeat check: every tracked page should contain some expected text (the competitor's name, a plan name) and alert you when it doesn't. Almost nobody builds this on day one. Almost everybody wishes they had by day ninety.

4. Where does the history live?

A diff only tells you what changed since yesterday. Most of the useful questions are about longer stretches: when did they drop the free tier, how many times have they renamed the mid plan, did the enterprise page change before or after the funding round?

If you only keep the latest snapshot, those questions can't be answered. Store every version with a timestamp from the start. It's cheap and it's the part you can't add later. The same goes for pages that are hard to re-fetch, like the feature flags hidden in a JS bundle, which can change several times a week.

5. What does it cost once you count the maintenance?

The weekend build is the cheap part. The real cost is the hour here and there: a new competitor to add, a selector to fix, a proxy that got blocked, a Slack channel everyone muted. Add that up over a year and compare it to the price of a tool. Be honest about who pays it. Usually it's your most senior engineer, doing it reluctantly.

This is the comparison we'd ask anyone to run before choosing Seeto, too. Seeto tracks the public surfaces you point it at (pricing, changelogs, careers pages, docs, integrations), keeps the full history, and turns diffs into discrete change events, so nobody on your team ends up owning a regex file. It won't read those changes for you or decide what they mean. That part stays with a person.

6. Who reads the output?

A monitor that nobody reads is a cost, not an asset. Before building, name the person who reads the alerts and what they do with them. It might be a PM updating a battlecard, or a founder deciding whether to respond to a price cut.

If you can't name them, don't build anything yet, script or tool. You'll just recreate one of the ways a competitor program quietly dies, only with more infrastructure.

---

Build it if you have a narrow question, one owner, and a heartbeat check. Otherwise the honest answer to "we could just write a script" is: yes, and then someone has to keep it working.