Scraping Firmographic Data From Company Websites at Scale

Firmographic data, the stuff that tells you a company's size, revenue, industry code, headquarters, and who's running the place, starts rotting the moment you collect it. That's the whole reason scraping this data at scale needs an actual pipeline instead of a script that hits a hundred websites and calls it a day. Get the sourcing wrong, and everything downstream (the extraction, the normalization, the refresh logic) just compounds the error.

Why firmographic data degrades so fast

Firmographic data describes a company as a business, not a person. Legal name, trade names, parent company. Employee count and revenue, standing in for how big the operation really is. Industry code (SIC or NAICS), headquarters and branch addresses, founding year, ownership type (public, private, backed by private equity, non-profit), and the people running each department.

None of that sits still. Sales teams use these fields to match accounts against an ideal customer profile, score leads, carve up sales territory, and run compliance checks before a deal closes. But companies hire in bursts, get restructured, move offices, get bought, get renamed. A snapshot taken in March is already partly wrong by June.

Revenue is the shakiest field of all. Private companies don't publish financials, so even after pulling from several sources and cross-checking, revenue estimates are around 65 to 75 percent accurate, a coin flip with slightly better odds. Private companies don't publish financials, so even after pulling from several sources and cross-checking, revenue estimates are around 65 to 75 percent accurate. That's a coin flip with slightly better odds.

Employee count behaves better, but only once a single source stops being trusted on its own. Pulling from three or more places brings accuracy to around 85 percent within a 20 percent margin, which is workable. Industry classification tells the same story: one source gets 72 to 80 percent accuracy, but a waterfall across multiple sources pushes that into the 93 to 97 percent range.

That gap, single-source in the 70s against multi-source in the 90s, is too wide to file away as a footnote. Get sourcing architecture wrong at the start and no amount of clever extraction or normalization downstream fixes it. It has to be the first decision, made before the pipeline's already limping. From here, four separate problems need solving: finding the right pages, pulling structured fields out of them, reconciling formats across thousands of sources, and knowing when a record has gone stale. Each one breaks in its own way.

Where firmographic signals live on the open web

Company websites come first, and for good reason. About pages carry founding details and mission language. Leadership or team pages name the people in charge. Careers pages hint at headcount and growth by way of open roles. Contact pages give the address list. It's information the company published itself, with no third-party lag and no question about whether it's fair game to look at.

No two company sites look alike, either. There's no shared template, no standard schema, just thousands of different layouts built by thousands of different teams with thousands of different opinions about where the "About" link should sit in the nav bar.

Structured secondary sources fill in what the raw website misses. LinkedIn company profiles carry employee counts, industry tags, size bands, founding year, headquarters, and open-role numbers, often clean enough to drop straight into a firmographic record. Crunchbase, Owler, and PitchBook add funding and ownership context. Business registries (Companies House in the UK, SEC EDGAR in the US, various state filing systems) offer legal names with high reliability. EDGAR holds up well for publicly listed companies, but private company ownership gets patchy fast once the source is a state-level filing.

Commercial data platforms are worth a mention here, though not as a replacement for scraping. ZoomInfo advertises a large catalog of company profiles with firmographic and technographic filtering. Apollo covers roughly 30 million companies. Both are useful for spotting holes in a dataset's coverage, but neither substitutes for scraping primary sources directly, because that's where the freshest, most specific detail actually sits.

The order of operations runs as a waterfall: start with the company's own site, layer in LinkedIn and registries, bring in commercial enrichment to patch remaining gaps, then resolve whatever conflicts are left standing. Each layer catches what the one before it missed.

Discovery at scale: building the URL corpus before a single field is extracted

Extraction is worthless if the scraper's pointed at the wrong page, or worse, never finds the right one. Discovery has to happen first, as its own problem, not a warm-up lap before the real work starts.

Four page types carry almost all the signal worth having. About or Company pages hold identity and founding details. Careers pages act as a rough proxy for headcount, location, and growth trajectory. Press or News pages surface funding rounds, acquisitions, and rebrands. Contact or Locations pages map the geographic footprint.

Building a corpus of URLs that lead to those pages usually takes a few approaches stacked together. A seed list plus crawler starts from known root domains, follows internal links, and classifies each page by content type using simple heuristics or a lightweight classifier model. Sitemap parsing works faster where it's available: most modern sites publish a sitemap.xml file, which hands over a full page inventory without crawling the site link by link. Search-based discovery queries a web search API directly for a company's About page ("site:example.com about," or similar), which tends to beat a full crawl when working across a large list of domains.

None of this scales by hand. Writing custom URL patterns one company at a time falls apart somewhere well short of a few thousand targets. The corpus has to be built and kept current programmatically, with a crawl-and-map layer that follows internal links, respects depth limits so it doesn't wander into the site's old blog archive, deduplicates what it finds, tags each page by intent, and hands the candidates off to extraction.

JavaScript rendering as a requirement for most company sites

Most modern websites don't hand over their content in the first HTTP response. What comes back instead is a mostly empty HTML shell, a handful of div containers, and script tags pointing at JavaScript files that build the actual page once it's in the browser. Company sites are no exception. React, Vue, and Next.js are the default framework choices now, and headcount widgets, office location maps, and embedded press feeds routinely load in after the initial fetch, not during it.

A plain HTTP request sent to a page built that way gets a response that comes back essentially blank. No headcount, no address, no press mentions. Extracting nothing works.

Layered on top of that is bot detection, and Cloudflare's footprint is wide enough that JavaScript rendering and anti-bot defense appear together almost every time. Cloudflare checks TLS fingerprints, runs its own JavaScript challenges, watches browsing behavior for anything script-like, and throws up a Turnstile CAPTCHA when a visitor looks automated. Running a plain Playwright or Selenium script without adjusting for any of that gets it flagged and blocked fast.

The rules keep shifting, too. Starting in mid-September 2026, Cloudflare began blocking mixed-use AI crawlers by default on ad-supported pages. That's a policy change with real teeth, landing on a market already expanding fast: bot security spending is projected to grow from roughly $1.27 billion in 2026 to $5.67 billion by 2034. With that kind of money flowing into detection, the old tricks (rotating a static IP, faking a header) stop holding up for anything running at real volume.

The practical floor: headless browser infrastructure configured to look like a real visitor, running through residential proxy rotation. One major bot mitigation vendor reportedly shipped more than 25 version changes to its detection system in a ten-month stretch. Nobody keeps up with that by hand-editing a config file. The only workable path is automated access infrastructure that adjusts itself. This is why managed scraping APIs that bundle browser rendering, proxy rotation, and unblocking into one call exist. They take that weight off a team instead of asking it to rebuild the wheel every quarter.

Extracting structured firmographic fields from heterogeneous pages

Writing a custom CSS selector for every About page across five thousand different companies is a slow-motion act of self-harm. It's a slow-motion act of self-harm. Every redesign, every A/B test, every minor template tweak breaks the selector, and someone has to go fix it by hand. This is exactly where handing extraction to an AI model earns its keep: instead of chasing layout changes, a model reads the page for meaning and adapts on its own.

The shift is from selectors to a data contract. Instead of telling a scraper which div to grab, the ask gets described in plain terms or as a schema: legal name, trade names, headcount or size band, revenue if it's stated anywhere, headquarters address, branch locations, founding year, ownership type, industry keywords, parent company. The system chains prompts, breaks the page into digestible chunks, and checks the output against that contract. No selector rewrite needed when the site gets redesigned over the weekend.

Reported figures on this kind of automated extraction put it at 30 to 40 percent faster than selector-based scraping, with accuracy reaching up to 99.5 percent, and it holds up even as the underlying page layout shifts. That's the gap between a pipeline that needs constant babysitting and one that mostly runs itself.

Output matters as much as the extraction itself. Results should come back as validated JSON tied to the firmographic schema, not loose blocks of text somebody has to parse by hand later. Tools like Pydantic or a JSON Schema validator enforce the typing right at extraction time. When a field comes back null or missing, a retry agent resubmits the prompt automatically, and a deviation log tracks how often the output format drifts, so nobody's blindsided months later when half the records are missing headcount.

Normalizing firmographic output across thousands of heterogeneous sources

Two companies can both report "employee count" and mean three different things. One site says "roughly 500 employees." Another gives a professional-network-style range, "501 to 1,000." A third says, in plain prose, "we employ over four hundred people globally." All of it needs to collapse into the same schema field, and that collapsing is its own stage of work, not a side effect of extraction.

A handful of fields cause most of the trouble. Industry gets described in free text, "SaaS," "cloud software," "enterprise technology," and needs mapping onto a fixed taxonomy, usually SIC or NAICS codes. Revenue appears as a range, in a foreign currency, tied to an unclear fiscal year, or is missing entirely for private companies, so the output needs a confidence score attached instead of one clean number pretending to be certain. Location gets written five different ways, "San Francisco," "SF," "San Francisco, CA," "San Francisco Bay Area," all pointing at the same headquarters, and untangling that takes address normalization and geocoding, not string matching. Headcount from a LinkedIn snapshot, a count of open job listings, and a self-reported range are three different measurements of three different things, and each has to carry its source and method as metadata rather than get quietly merged into one number. Ownership type is rarely stated directly. It gets inferred from phrasing, "our investors include," "publicly traded on Nasdaq," "employee-owned," which is a judgment call more than a lookup.

Language models are well suited to exactly this kind of judgment call: classifying loose industry text, sorting out location variants, reading ownership language for what it implies. It runs on the same model infrastructure as extraction, but it's a different task with a different way of failing.

And when sources disagree, which they will, the pipeline needs a standing rule for who wins. Company website says 200 employees, LinkedIn says 500 to 1,000. The pipeline needs a standing rule for which source takes precedence, with both source values kept in the metadata rather than thrown away. Whatever the rule, it has to be decided in advance, not improvised record by record every time two numbers disagree.

Keeping firmographic records fresh at scale without re-scraping everything

Re-scraping fifty thousand company pages every week on a fixed schedule burns money and mostly re-downloads data that never changed. The smarter move is refreshing records only when something's actually shifted. This means detecting change before deciding to re-scrape.

A few methods handle that detection. Page-level monitoring hashes or diffs the cleaned content of key pages (About, Careers, Press) on a set schedule and flags anything past a set threshold for re-extraction. Signal-based triggers catch it earlier: a spike in job postings hints at a hiring surge and a headcount change, a new press release hints at funding or an acquisition or a rebrand, a jump in LinkedIn followers hints at growth. Each is a cheap early warning that firmographic data is about to move. External event feeds, funding databases, business registry updates, news APIs, act as upstream triggers too, catching a change before the company's own website ever reflects it.

Not every field needs the same refresh clock, either. Headcount, open roles, key personnel, and recent funding move fast enough to warrant monthly checks or trigger-based updates. Revenue range, office locations, and ownership structure shift slower, so quarterly is usually enough. Founding year, legal name, and industry classification barely move at all, so annual checks, or checks only when something else flags a change, do the job.

A content hash cache at the crawl layer closes the loop. If a page hasn't changed since the last crawl, there's no reason to burn extraction credits or LLM tokens reprocessing it. Skipping unchanged pages means not paying twice for data already sitting on the shelf. It's just not paying twice for data already sitting on the shelf.

Putting the layers together: what a production firmographic pipeline looks like end to end

Once the specifics are stripped away, a working firmographic pipeline comes down to five parts, stacked in order, each one handing cleaner input to the next.

Discovery builds the URL corpus, using search APIs and sitemap parsing to find the About, Careers, Press, and Contact pages worth scraping. Access handles the technical fight (JavaScript rendering, headless browsers, proxy rotation, unblocking) usually through a managed scraping API rather than infrastructure built and maintained in-house, since keeping pace with anti-bot updates by hand isn't a realistic long-term plan for anyone. Extraction pulls structured fields out using a defined schema and intent-based prompts instead of brittle selectors, with retry logic catching null fields before they turn into a gap in the record.

Normalization takes whatever mismatched output comes out of extraction and standardizes it: mapping industries to a taxonomy, geocoding locations, binning headcount ranges, scoring confidence, resolving conflicts between sources according to rules set in advance. Freshness closes the loop, using content hashing and signal-based triggers to catch which records have gone stale and need another pass, instead of re-scraping everything on a blind schedule.

None of these five layers is optional, and none works well on its own. Extraction has nothing reliable to work from unless discovery happens first. Without normalization, every downstream report ends up comparing numbers that were never speaking the same language to begin with. Without freshness, the whole system slowly turns into an expensive archive of facts that used to be true.