Method
- Home
- Method
How two portals become one catalog
one source publishes the files, the other source publishes the statistics, and both classify by the same 17 statistical fields. Four passes turn that into one index.
Nothing below is inference about meaning. Each pass is a rule applied to a field one of the portals already publishes, and where a portal publishes nothing, StackData records the absence instead of filling it in.
-
01
Harvest
Read both APIs on their own terms: some open, some keyed and paged.
-
02
Normalize
Collapse licence spellings and cadence wordings to one value, keeping the raw string.
-
03
Classify
Join on the 17 shared fields, and leave the uncategorised uncategorised.
-
04
Verify
Re-fetch, flag series that stop, and try to see one statistic once.
01 — Harvest
Two APIs that agree on almost nothing
Some portals run a standard catalog platform whose search endpoint answers without a key, sometimes at an address the portal's own documentation never mentions. Finding those is part of the work, and a single count request returns the size of what is there before anything is downloaded: 18,144 datasets across 23 publishers, read in August 2026.
Others are gated, requiring a registered key on every call, and answer a failed request with a success code and an error inside the body rather than an HTTP error — so a harvester that trusts status codes silently collects nothing. Paging is bounded as well, capped per request and continued through a cursor. Where a source offers several endpoints, StackData harvests through whichever one covers both its published files and its database tables, which is rarely the obvious one.
Where no request limit is published, the terms usually still reserve the right to throttle under load, so the harvester runs slowly by default and keeps every credential out of anything it ships.
| Attribute | Open API | Keyed API |
|---|---|---|
| Key required | No | Yes, on every call |
| Unit of record | Catalog dataset | government statistic, 8-digit code |
| Licence lives on | Each resource | Each resource, plus one site-wide rule |
| Second classification level | None | 58 subfields |
| English metadata | None for datasets | Survey level, a small minority |
02 — Normalize
One licence, four spellings, in one portal
one source's licence field is a controlled vocabulary only on paper. its licence vocabulary publishes 14 keys, three of which name the same licence family: cc-by, cc-by-4.0, cc-by-2.1-jp. Records then reach past the list entirely. jinji_20211001_0008 stores CC-BY, uppercase and hyphenated, matching no key. maff_20180312_0467 stores Government Standard Terms of Use 2.0, the human label of gjstu pasted into the id field. mlit_20140919_0726 carries both spellings at once, on different resources of a single dataset.
Cadence fails the same way. one source's update field is free text, so one annual cadence arrives as “1 year”, as the same string typed with a different digit character, as bare “year”, as “at least once a year”, and as an empty string — all on the same portal. “Irregular” and “not updated” sit in the same field, and neither is a frequency. the other source instead uses a short fixed list, with a dash for statistics that declare none. StackData maps both onto one cadence value and keeps the published string on the record, so the mapping can be checked against what the portal actually said. Raw strings below are shown in translation.
Licence and cadence, as harvested
one source resource fields
Raw
Normalized
- cc-by
- CC-BY
- cc-by-4.0
- cc-by-2.1-jp
- gjstu
- Government Standard Terms of Use 2.0
- “1 year”
- “1 year”, different digit character
- “year”
- “at least once a year”
Values quoted exactly as the source API returns them, August 2026.
03 — Classify
The join the publishers already made
Both portals classify by statistical field classification, the same 17 government fields, and the 17 labels are string-identical on each side. There is no crosswalk to invent, which is the reason one index over two portals is possible at all.
One trap sits at the end of the list. the other's codes run 01 to 16 and then jump to 99 for Other, even though the other's own table numbers that row 17. A join on position breaks there, so StackData joins on the label and maps 99 explicitly. one source exposes the labels and not the codes, so the match rests on the strings. the other source also carries a second level of 58 subfields; one source has none, so the shared axis stops at the top.
04 — Verify
Re-fetch, then decide what stopped
Every record is fetched again on a schedule. Where a source accepts an updated-since filter, that is the feed StackData follows, because it is far cheaper than re-reading the whole catalog. A portal’s human-facing “what’s new” page is rarely a substitute: they routinely cover one layer of the holdings and not another, and say so in documentation nobody reads.
When a re-fetch turns up nothing new for longer than a statistic's own cadence, StackData marks the series stopped rather than leaving it looking current. The mark is a claim about publication, not about the survey, and it stays provisional: the Housing and Land Survey falls from 13,440 files for 2013 to 684 for 2018 because publication practice changed, not because the survey did.
Deduplication is the hard pass. One in four datasets in the index is hosted by a different source than the one that lists it, so the same statistic is genuinely present twice. Titles cannot join them: of 4,533 such records, exactly 12 titles matched their counterpart and 3,572 matched nothing, because one catalog flattens the other's hierarchy with inconsistent separators and writes the year sometimes in a local era calendar and sometimes as a Western year, inside a single survey. Identifiers do join — but they arrive under twelve different parameter names.
The Monthly Labour Survey is the plain case: two records, two unrelated identifiers, one survey. The reverse happens too — the Family Income and Expenditure Survey appears on one source and not the other — which is why any single portal, read alone, loses real statistics.
One key, twelve parameter names
12 parameter names, 2 key spaces
Raw
Normalized
- statInfId
- sinfid
- stat_infid
- toukei
- tstatCode
Counted across the 4,533 cross-hosted datasets. Seven further spellings of the same key appear beyond those shown.
Scope
What StackData does not do
- No full-text search
- You can browse by category, publisher and source. You cannot search inside titles or descriptions. Source-side keyword search is often loose enough that a national census query returns an unrelated working paper first, and a search that behaves that way is worse than none.
- No public API
- The catalog is static HTML. Nothing here is queryable by machine, and there is no endpoint to point a script at.
- Two sources only
- Two national portals. Regional and municipal portals are not indexed yet, and neither is anything published outside them.
- Dedup unresolved
- StackData can see the 4,533 overlapping datasets and can normalize the twelve identifier spellings, but the join has not been run end to end against every resource URL. Until it is, one statistic can appear twice and both rows are shown as they arrived. Duplication exists inside a single source too: 4,916 distinct table ids are referenced by more than one one source dataset.
Next, in order
Run the identifier join across every source's resource URLs and collapse duplicates at the statistic level. Resolve licence per resource rather than per record, since that is where portals actually store it. Then full-text search over titles, once there is a single title per statistic to search.