What We Leave Out of Our Job Index (And Why We Deleted 5,000 Rows That Made Us Look Bigger)

Our database holds 332,693 open roles this morning. The public dataset we publish ships 247,576 of them. The missing 85,117 are not a sync failure. They are five rules, and every one of them deletes rows we could otherwise have counted.

Two of those rules are boring. One — the filter for job postings from staffing agencies — turned out to be much harder than it looks, and we got it wrong twice. One of them removed 5,134 rows in a single commit, purely because a sentence on our dataset card was false while they were in there.

This is what each rule does, why it exists, and what it costs. Numbers are from the live database, measured while writing this.

Five clauses, one function, no per-endpoint exceptions

The rules live in a single function that every query importing companies has to call. That sounds like trivia, but it is the load-bearing part. The previous version was a per-request option that defaulted to off, plus a hand-rolled domain check buried in the search pipeline. Which meant the answer to "is this company visible" depended on which endpoint you asked and what the search form happened to send. Two of our own pages disagreed with each other for weeks.

The five clauses, in the order they run:

1. There is somewhere to apply. A crawled careers page, or an open role carrying its own working link.
2. There is an open role. A company with a careers page and nothing on it is a dead end with good manners.
3. The domain is commercial. No .org, .edu, .gov.
4. The name is not an agency's name. The interesting one; see below.
5. The ATS board is not on a small blocklist. Also see below.

Rule 3 has a hole worth naming. A NULL domain passes. That is deliberate: 5,685 of our visible companies have no resolved domain at all, because they arrived through ATS-board harvesting rather than a web search, and there is no second field to fall back on. Hiding three-quarters of the index over a domain we failed to resolve would be measuring our own gap, not a fact about the employer. So the honest statement of rule 3 is: when we know a company's website we refuse non-commercial ones; when we do not, we show the company.

Why job postings from staffing agencies are hard to filter out

Agencies are a small fraction of employers and a large fraction of listings. Right now, 301 companies in our index have an agency word in their name — about 1.2% of the 24,999 companies with anything open. Those 301 hold 6,050 open roles, or roughly 20 each, against an index-wide average of 13. A staffing firm posts more, reposts more, and never stops.

The filter is a list of tokens matched at word boundaries: staffing, recruiting, recruitment, headhunt, zeitarbeit, personalvermittlung, talent, outsourcing, workforce, manpower, and a dozen more. Word boundaries rather than substrings, because a bare substring test reads "SmartRecruiters" as a recruiter — it is an ATS vendor and a real employer that hires real people — and reads "MongoDB", "Duolingo" and "Tango" as non-profits.

The two entries that are not on the list are the more useful ones. We measured agency and it matched 16 companies including OLIVER Agency and Human Agency, both of which hire for themselves. We measured consultancy and it matched Elmwood Brand Consultancy, a design firm. Both rejected. The rule that fell out of it: an agency word is only safe when it describes the trade of supplying people, not the trade of doing work. "Staffing" always means the first thing. "Agency" means either.

Then there are the agencies whose names give nothing away. Hyphen Connect Limited is a Web3 recruiter with 1,027 open roles across completely unrelated employers — a CMO, a compliance officer, a DeFi product owner. There is no token in that name to match on without also matching real companies. Uptalent.io runs the word together so no word-start token reaches it. So those are keyed on the ATS board slug instead, which is the stable identifier: a company can rename itself, but its board URL is where its postings actually live. Three slugs are blocked. They hold 2,727 open roles between them.

That blocklist stays short on purpose. If it starts growing, the answer is a better signal, not more entries in a junk drawer.

The 5,134 rows we deleted from the public dataset

We ingest three public employment services: Sweden's Arbetsförmedlingen, Germany's Bundesagentur für Arbeit and France Travail. They are good sources. They are also, by design, public feeds with open APIs that any aggregator can read, and several dozen do.

The dataset card said every row was "read directly from each employer's own careers page and public ATS feed — never aggregated or reposted from a job board." With the state boards in there, that sentence was false for 5,134 rows.

We took them out. The published count went down. It has kept going down in relative terms, because those feeds have grown to 16,473 open roles since — all of them still in the product, none of them in the public file.

There was a second reason, visible the moment you sort those feeds by employer. The biggest posters are Akkodis Germany Tech Experts (195 roles), puro Personaldienstleistung (140), DIS AG (116), ACADEMIC WORK SWEDEN (102). The public boards are where agency volume concentrates, which is not a criticism of the boards — placing people is what a national employment service is for — but it does mean the aggregate is not what a reader assumes when they see 5,000 more rows.

The cost is real and we should state it rather than let someone find it: a labelled subset of those rows would have been genuinely useful for anyone studying European labour markets, and right now there is no way to get it from us. An unlabelled subset that contradicts the card was the worse option, so it went first. The labelled one can come back.

Why there are no job descriptions and no apply links

Two separate decisions that people assume are one.

Descriptions are a copyright question. Under Feist v. Rural Telephone (1991), facts are not copyrightable no matter how much work went into collecting them — the Supreme Court threw out the "sweat of the brow" doctrine outright. That a company posted a Senior Backend Engineer role in Berlin on 14 August is a fact. The 700 words the hiring manager wrote describing it are authored text, and they belong to whoever wrote them. So we publish the facts and not the prose. The compilation is CC BY 4.0; the underlying listings stay the employer's.

Apply URLs came out for a different and less noble reason, and it is worth being straight about which is which. A posting URL contains the employer's ATS token. So does the careers-page column. So the URL column is not one field — it is the company-to-ATS-board mapping, which is the output of the whole harvesting pipeline and the one asset here that does not go stale. Job rows die in days. The board list does not. Publishing it would let anyone skip the discovery problem and stand up a live index in an afternoon.

That is self-interest, not principle, and the dataset card says so in the limitations section: without a URL, a row cannot be de-duplicated against another dataset or followed back to the posting. If you are doing research that needs the link, this dataset cannot help you, and you should know that before you download 38MB.

We also hold back roles found by scraping a company's own careers page rather than reading an ATS feed. That is the half of the index no aggregator has — it is the 18% we measured that never reaches the feeds job boards ingest — and it stays in the product.

What the exclusions leave you with

A labour-market dataset: who is hiring, for what, where, and when it went up. Not a job board, and the card should not pretend otherwise. Company names come from ATS tokens for board-sourced employers, so some render as slugs (Openai rather than OpenAI). Locations are free text exactly as published, so London, London, United Kingdom and London, GB all occur and you have to normalise before you aggregate. posted is missing on a minority of rows because not every feed publishes it.

None of this makes the number bigger. Every rule above is a subtraction, and the one that hurt was the one where the deleted rows were fine — they just made a sentence untrue that a stranger could check. A count you can't check is a marketing number, and the whole point of publishing the file is that somebody can go and check.

The full index is on Hugging Face and a browsable daily snapshot, sliced by city and role family, is on GitHub. Both free, no account. If you find an agency we missed, open an issue — that is how three of the tokens above got added.

FAQ

Are job postings from staffing agencies worth applying to? Sometimes, but you should know which one you are looking at. An agency posting is a lead-generation ad as much as a vacancy: the same role may be advertised by four agencies at once, the description is often deliberately vague about the actual employer, and "we'll keep you on file" is a real outcome. If you want to apply to a company, apply on that company's careers page.

Why does the same job appear on ten different job boards? Syndication. A single posting propagates through feed partnerships and programmatic advertising across a market with, by LinkUp's count, more than 25,000 job boards. Most of them do not originate anything; they backfill from each other. It makes freshness dates unreadable, because the date you see is when that board got the row, not when the employer posted it.

Can anyone republish job descriptions? The facts about a listing — title, employer, location, date — are not copyrightable, per Feist. The description text itself is authored work and belongs to whoever wrote it. That distinction is why our public file has the first set of columns and not the second.

Where can I download an open dataset of job postings? Ours is at huggingface.co/datasets/Lynceus/jobs under CC BY 4.0. Read the limitations section before you use it for anything — no apply URLs, unnormalised locations, and a coverage skew toward companies on Greenhouse, Lever, Ashby and Workable.