LLM-Powered Structured Extraction From Unstructured Product Pages
Semantic extraction survives redesigns while selector-based systems crumble under maintenance costs.

This article is about why product data teams are dropping CSS selectors and XPath in favor of LLM-powered extraction, and why that switch is less a style upgrade than a survival move. Selectors work by pointing at an address: a class name, a tag, a spot three levels deep in the page's nesting. That address only exists because a developer built the page a certain way on a certain day. Changing the page erases the address, along with every rule anyone wrote against it.
React and other modern frameworks make this worse, not better. Class names get generated on the fly, so the "address" can shift between one page load and the next, with no redesign required. A selector that worked at 9am can fail by lunch, for no reason anyone on the team did wrong.
That's the trap: the cost of this approach isn't a one-time setup fee, it's rent, paid every time a site team ships a change. None of that work produces new data. It just keeps old data flowing, until the next redesign resets the clock.
How semantic extraction replaces structural pattern-matching
LLM-powered extraction fixes this by giving up on addresses. Instead of asking "where in the HTML does the price live," it asks "what counts as a price," and finds the answer wherever it happens to sit on the page. A model told what price data looks like can find it whether it's in a div, a span, or buried in some new component a front-end team invented last Tuesday. The page can be rebuilt top to bottom, and the instructions describing what to look for still hold.
That's a different kind of durability than "slightly more resilient selectors." A redesign that would snap a CSS rule in half can't touch a semantic description, since the description was never tied to markup to begin with. Giving someone directions by street address versus describing the building shows the contrast: paint the building a new color, put up new signage, and the second set of directions still gets you there.
The shift also changes who's allowed to write extraction logic. Natural-language extraction instructions can be written by a product manager or a data analyst who has never opened a browser's developer tools. That means extraction logic, instead of being locked behind a technical skill most people don't have time to learn, can live closer to the people who actually understand what the business needs from the data.
Research presented at SIGIR '26, held July 20 to 24, 2026, in Melbourne, puts a number on how far this generalizes. Meaning-based extraction doesn't just survive a redesign. It survives a catalog with thousands of categories that were never going to share a consistent page layout to begin with.
The four stages of a production extraction pipeline
Here is where a lot of teams get the wrong idea about what this work actually involves: they picture one step, "call the model," and build everything around that assumption. In practice, a production pipeline has four stages, and the model only shows up at the last one. Fetch and render are where most pipelines actually fail, long before an LLM ever sees a product page.
Fetch comes first, and it's less trivial than it sounds. A meaningful share of product pages, something like 20 to 25% according to an analysis of web extraction infrastructure, won't show their content until a browser runs the page's JavaScript. Skipping this stage or underbuilding it causes the pipeline downstream to receive empty pages or block screens, which then gets misdiagnosed as a model problem when it's actually a plumbing problem.
Render comes next. Skipping rendering by default leaves a quarter of the catalog incomplete.
Clean is the third stage, and it's the one teams skip most often, usually to their regret. A team that skips this step pays, page after page, for tokens that describe a cookie banner.
Only at the fourth stage does the model actually do anything. By the time extraction happens, the content has already been fetched, rendered if needed, and cleaned down to what matters. The model's job here is narrow: read a prompt describing what product attributes to find, and fill in a schema that defines what the output should look like. It isn't crawling the DOM or rendering pages; it's figuring out what a sentence means.

Why the output format is not a cosmetic choice
The format the extracted data comes out in decides whether the system reading it afterward can trust the answer or has to guess at it. The other is a fact.
The SIGIR '26 attribute graph research shows what this buys in practice. Extraction is instructed to produce strict JSON, so the ranking model that uses that output reasons over a curated set of attributes. Structure didn't just make the output easier to use. It made the whole system cheaper and more accurate at the same time.
The practical move is to write the schema before writing the extraction prompt, not after. The schema is the spec the model has to fill in, and locking it down first forces the extraction task to be precise from the start, rather than hoping something structured falls out of a loose paragraph of model output after the fact. In Python, a Pydantic schema is what actually enforces that spec at runtime: it catches a missing field, a price that came back as text instead of a number, or a value the model invented, before any of those errors get passed downstream into a pricing engine or an inventory system where they're far harder to spot and far more expensive to clean up.
This isn't a theoretical concern. ExtractBench, a benchmark released in July 2026, tested schema-guided extraction across hundreds of enterprise documents, thousands of pages, dozens of document types, and multiple business domains, scoring systems on value accuracy, how complete the records came out, grounding, and cost, all at once. The point of a benchmark like that is to establish a baseline everyone gets measured against. Schema-guided extraction is that baseline now, not a nice-to-have bolted on top of a simpler approach. Schema validation catches specific failures before they ever reach production, including rows quietly dropped from a long product list, a sale price mistaken for a list price when both appear on the same page, and a field overstuffed with text that belongs somewhere else.
Cleaning and schema design as the fix for the token cost trap
Running this pipeline at scale comes with a cost curve that punishes sloppiness fast. Feeding a model raw HTML means most of what it's charging for is navigation menus and script tags, not product information, and that waste multiplies with every single page the pipeline touches.
Two decisions control that cost more than anything else, and they work in sequence. A cheaper, value-tier model working on clean text can do the extraction for a fraction of what a top-tier frontier model would cost working on raw, uncleaned HTML.
Schema design adds a second layer of savings on top of cleaning. When a model reasons over a short, structured list of attributes instead of a full paragraph of marketing copy, it simply has fewer tokens to read per product. That 57% token reduction from the SIGIR '26 work isn't a side effect, it's the direct, intended result of handing the model a tighter, more structured input.
Volume also changes which architecture makes sense. At low request volumes, running the full LLM pipeline on every page is fine, even without squeezing every bit of optimization out of it. At very high volumes, a hybrid setup tends to win out: LLM extraction reserved for pages that are messy or constantly changing, and a simpler, deterministic extraction method for high-volume pages that follow a stable, predictable layout. Pipeline hygiene, in other words, is a budget line.
The SIGIR '26 attribute graph architecture and extraction at catalog scale
The SIGIR '26 attribute graph system is a working example of this entire pipeline running at the scale of a real product catalog, and it splits its work into two separate stages, offline and online, which is the whole design.
Online, when a search or ranking request comes in, the system does not go back and re-read the original messy text. A graph-aware ranking model reasons directly over the structured attribute-value pairs already sitting in the graph. That's faster, cheaper, and more precise, because the model is working with facts.
That split is the fetch-clean-extract pipeline, applied at catalog scale. Do the expensive semantic work once, store it as clean structured data, and every later operation that touches that product gets to skip the noisy re-reading step. The 57% drop in per-product tokens is the direct payoff of designing the schema well in the first place, and that payoff appears every time the data gets used afterward.
Building the pipeline in practice: fetch, render, clean, and prompt as connected decisions
None of these four stages operates on its own. They're linked, in order: how a page gets fetched decides what rendering it needs, how it gets rendered decides what cleaning has to happen, and how it gets cleaned decides what the extraction prompt has to handle. Treating them as four separate boxes to check, rather than one connected chain of decisions, is where a lot of pipeline designs go wrong before the model ever enters the picture.
On the fetch side, the right approach depends entirely on what kind of page is in front of the pipeline. A static product page just needs a plain HTTP request, and firing up a headless browser for it adds cost without adding anything useful. Pages that actively try to block automated visitors need something else entirely: proxy rotation, careful handling of TLS fingerprints, retry logic built to survive a block, and none of that belongs in application code. It belongs down in the fetch infrastructure itself, where it can be built once and reused everywhere.
Once a page is fetched and rendered, cleaning strips out the navigation bars, scripts, ad placements, footers, and sidebar junk, leaving only the actual product content behind. Some web data APIs now return clean markdown or structured content directly instead of raw HTML, which quietly removes cleaning as a custom engineering problem a team has to solve for itself, and hands the model clean input from the start.
The schema and the prompt are the last link in the chain, and they need to be built in that order, not reversed. The JSON schema for the product data should exist before a single word of the prompt gets written, because the schema is the spec the model is filling in, and settling it first forces the extraction task to be precise from the outset. Pydantic, or an equivalent validation tool, enforces that schema at runtime, catching a missing field or a mismatched type the moment extraction happens rather than after it's already landed somewhere downstream. The model's job is handling layout variation across pages. The prompt's job is handling the ambiguity in what the data actually means.
Schema-guided extraction as the standard for enterprise and AI-agent workloads
Schema-guided extraction, where a schema someone actually wrote drives the extraction and the output comes back as validated JSON with grounding attached, has moved from "advanced technique" to standard practice for extraction work done by AI agents, not just a trick a few teams happen to use.
Commercial vision-language models tend to do fine on short documents but start dropping records once a document runs long. For a product catalog, that risk appears in pages with a lot of SKU variants, long spec tables, or attributes spread across multiple paginated screens. Any extraction system that doesn't handle record completeness on purpose will quietly lose data on exactly those pages, the ones with the most information worth keeping.
Grounding is the audit trail of knowing exactly which part of the source page a given value came from, and it's what lets a pricing engine, an inventory system, or a downstream AI agent act on a piece of extracted data without a person checking every single record by hand.
That matters just as much for AI agents as it does for batch jobs. An agent working over the Model Context Protocol, whether it's a Claude instance or an automated pricing bot, needs the exact same clean, schema-validated JSON a scheduled batch pipeline produces. The underlying extraction infrastructure has to serve both: a person running a nightly job and a software agent making a live request, off the same pipeline, with the same guarantees.
Teams that put real care into their schemas, validate the data the moment it's extracted, and invest properly in fetch infrastructure end up with something durable: pipelines that keep working as target sites change shape, that scale to large catalogs without costs climbing in a straight line, and that produce data AI agents can act on directly, no human double-check required. That combination is what AI-ready product data actually means in practice.


