How to Detect When a Website Change Has Broken Your Data Collection

Why scrapers fail silently

Website changes do not always produce a clear error. A redesign may leave the page available while changing a CSS class, moving a field, renaming a label or returning a consent page. The scraper completes its loop, but the output contains blanks, duplicate values or the wrong content.

This is more dangerous than a visible crash because incomplete data can flow into a dashboard, sales list or pricing report without anyone noticing. Reliability starts with treating the output as something that must be tested, not merely a file that must be created.

The signals that reveal a broken collection

Use several independent checks instead of relying on one selector or exit code. A source can change in a way that passes one check and fails another, so combine volume, structure, content and business-rule validation.

• Record count falls outside an expected range.
• Required fields become empty or contain the same repeated value.
• A known page no longer contains its expected heading or marker.
• The percentage of duplicate URLs or identifiers suddenly rises.
• Dates, prices or categories fall outside reasonable business ranges.
• The response is a login, consent, error or challenge page instead of the target content.

Build validation around the business output

The most useful rules describe what a good dataset looks like. For example, a daily product collection may require at least a minimum number of records, a non-empty product name, a valid price and a unique source URL. A lead workflow may require a company, source page and a review status.

Keep thresholds configurable because different sources have different normal ranges. When a check fails, preserve the raw response or diagnostic sample and mark the run as needing review instead of publishing it as normal.

Monitor source changes over time

Save a small history of run metrics: start and end time, pages visited, records extracted, missing-field counts, duplicate counts and validation failures. Comparing these values with recent runs makes a gradual source change visible.

For important sources, store a sanitized page fingerprint or a few representative snippets. This gives the maintainer evidence about what changed without retaining more content than the workflow needs.

What to do when a scraper breaks

Pause downstream delivery when the failure affects a required field or a large part of the dataset. Review the last known-good run, compare a current page with a saved sample and identify whether the source changed, access was restricted or the workflow reached an unexpected page.

After the fix, replay a small sample, compare the new output with the previous run and record the change. A short maintenance note prevents the same investigation from being repeated later.

• Alert the owner with the failed rule and affected source.
• Keep the last known-good output available for reporting continuity.
• Test the fix against normal, missing and unusual page states.
• Reconcile record counts and key values before resuming delivery.
• Document the cause, fix and date of the source change.

How AI Scraping Lab can help

AI Scraping Lab can design monitored extraction workflows with structured outputs, validation rules, exception reports and scheduled delivery. The goal is to make website data useful and maintainable, not simply to collect a large number of pages once.

A discovery call can start with a few representative URLs, the fields you need, the required refresh schedule and examples of records that must be rejected or reviewed.