Data aggregator
platform development
in Westwood

Every source describes the same kind of item a little differently. One feed calls a field price and sends a plain number. Another calls the same field cost and sends a formatted string with a currency symbol attached. A third splits price into a base amount and a separate fee line. None of this is wrong from the point of view of the source. It just was not built with any other system in mind.

A normalization layer exists to absorb that mismatch before anything reaches a buyer-facing screen. Every incoming record gets mapped onto one internal schema, regardless of how the source chose to structure its own data. Every field gets converted into one consistent unit, format, and type. A price is always a plain number in one currency by the time it leaves this layer, never a string that only looks like one.

Categories cause a similar headache. One source calls a listing plumbing services. Another calls the same kind of business home repair. A third buries it three levels deep inside a general contractor category with no clean equivalent anywhere else. Building one internal taxonomy and mapping every source category onto it, even messily at first, beats showing a dozen inconsistent labels for what is really one topic.

Missing fields need a policy, not a guess made field by field as problems appear. Some sources omit a field entirely rather than sending it empty. A normalization layer has to decide whether a missing field means zero, means unknown, or means the listing gets held back from display until a person reviews it. Silently treating a missing field as zero can quietly distort a ranking or a filter for months before anyone notices.

Source schemas change without warning far more often than teams expect. A partner renames a field, splits one column into two, or quietly starts sending a null where a number used to sit. Validating every incoming batch against the expected schema, and flagging a batch that fails validation instead of loading it anyway, keeps one upstream change from corrupting the aggregated dataset a user actually sees.

Testing a normalization pipeline differs from testing typical application code, because the input is a moving target owned by someone else. A useful test suite keeps a small library of real, slightly odd sample records pulled from each live source. Some carry a strange price format. Some carry a missing category. Each change to the pipeline gets checked against the whole library, messy records included alongside the tidy ones.

Done well, normalization is invisible. A buyer scrolling through listings from a dozen sources should never be able to tell which source a given row came from just by the shape of the data on screen. That invisibility is the entire point. It is also why the work behind it rarely gets the attention it deserves until it breaks.

Scraping a public page is fast to build and free to start. A script visits a page, reads the structure, and pulls out the fields a platform needs. No conversation with the source required. That speed is exactly why teams reach for scraping first and worry about the consequences later.

The consequences show up soon enough. A source can change its page layout without notice. Every scraper reading it breaks overnight. A source can add a login wall. It can add a rate limit. It can add bot detection aimed squarely at stopping automated collection. A scraping approach has no recourse beyond building yet another workaround.

Terms of service matter here in a very practical sense, apart from any legal question. A source that notices heavy scraping traffic and dislikes it can block an entire hosting provider rather than one script alone, which can take down other unrelated systems sharing that infrastructure. Reading the terms before building a scraper, and keeping request volume modest and clearly identified, avoids the kind of blunt response that hurts far more than the one integration involved.

A feed agreement or an API partnership solves most of these problems. The cost is time. Often it is a share of revenue or a flat fee too. A source that provides a structured feed agrees to keep a stable contract in place, version it deliberately, and give notice before breaking changes land. That stability is worth paying for once a data source becomes central to what a platform shows users every day.

The decision usually comes down to how central a source is, not how easy it is to reach. A minor source adding a small amount of long tail coverage rarely justifies the cost and delay of a formal agreement. Light scraping, done politely, is a reasonable stopgap there. A source that anchors an entire category deserves a real conversation with the business behind it, well before that source becomes indispensable. Wait for trouble first, and the conversation starts from a position of weakness.

A hybrid path works for many platforms during a transition. A team can scrape a source lightly while a partnership conversation is underway. The scraped feed acts as a bridge, not a permanent fix. The team switches over cleanly once a formal feed is ready. Being upfront with a prospective partner about the interim scraping, rather than hiding it, tends to make that later conversation easier.

Whichever path a platform takes for a given source, the integration should be built so swapping the underlying method costs little. If the internal schema and normalization layer do their job, moving from a scraped feed to a licensed one changes nothing downstream except the reliability of what arrives. A buyer looking at the results should never notice the switch happened at all.