Engineering the pipeline

This project publishes fresh forecast profiles for every catalogued site, every model run, around the clock — and its total infrastructure is a public GitHub repository. No server, no database, no API keys, no bill. This page records the patterns that make that work, because each one is reusable well beyond windgrams.

GitHub Actions as the compute plane

The entire pipeline is one workflow on a public repository, where Actions minutes are free. A schedule fires every 15 minutes; each builder asks its upstream "is there a run newer than the one in data/?" and exits if not. A no-new-run check costs a single request, so polling aggressively is polite. When a run has advanced, the builder fetches, derives, writes to data/, and the workflow commits.

A concurrency group serialises builds so two scheduled fires never race, and experimental legs are wired to warn instead of fail — a dark HRDPS 1 km feed must never block the 2.5 km publish. As more model builders land, each is its own step with the same contract: exit quietly on nothing-new, write to your own directory on success, fail loudly on anything malformed.

Git-scraping: the repository is the database

Every build that finds a new model run ends in exactly one commit, made by the workflow itself. That single decision buys a lot:

• History is free. git log data/manifest.json is the pipeline's operations dashboard: when each run landed, what it cost upstream (the manifests embed request counts, bytes, and durations), and what changed.
• Diffs are the audit. The derivation writes keys in a stable order and serialises deterministically, so a republished run diffs cleanly and any unexpected change is visible in review.
• Rollback is git revert.

raw.githubusercontent.com as a zero-config CDN

Consumers read the published JSON straight from raw.githubusercontent.com/azohra/windgrams/main/... — no key, no server of ours, backed by GitHub's CDN. Two properties matter in practice:

• The cache is ~5 minutes. Fine for hourly-at-best model runs.
• Files are cached independently, so a consumer can briefly see a new manifest with an old site file (or vice versa) while the CDN converges after a publish. The defence is in the data: every profile embeds its own referenceTime, so a consumer compares it against the manifest's and simply keeps its previous copy when a fetch comes back from a different run — a reference-time skew guard, a few lines on the client, no server-side coordination at all.

Fetching kilobytes from gigabyte files

The single most important cost trick in the pipeline: never download a model field you don't need.

• GRIB .idx byte-range subsetting. NOAA (and others) publish a plain text .idx sidecar next to every GRIB2 file listing each record's name, level, and byte offset. HTTP range requests then pull exactly the records needed. A single HRRR pressure file is ~546 MB; the 54 records a windgram hour needs — all of which live in that one file — cost kilobytes to a few megabytes. One .idx fetch, one ranged fetch per record (or coalesced ranges), done.
• GeoMet WCS crops are cheaper still. Where ECCC's GeoMet offers a model over WCS, the server does the subsetting: a GetCoverage request for a small box around the sites returns a ~2 KB GeoTIFF. The full 2.5 km HRDPS build moves ~6 MB total. The GeoTIFF reader (windgrams/geotiff.py) is deliberately minimal — uncompressed float32 strips only — because anything fancier arriving from upstream is a change worth failing loudly on.
• Whole files are the last resort. The experimental HRDPS 1 km leg has neither WCS nor .idx sidecars, so it downloads ~1.2 GB per run to sample four launches. That contrast — 6 MB vs 1.2 GB for the same catalogue — is the argument for subsetting in one line.

One manner worth copying regardless of transport: clients identify themselves with a real User-Agent pointing at this repository, retry 429/5xx with jittered backoff, honour Retry-After, and publish their request counts in the manifest. Free data stays free when consumers are visible and polite.

Probe for completeness before building

Model runs appear on servers file-by-file over many minutes. Fetch eagerly and you build from a half-published run. Two guards, used by every builder:

• HEAD the last forecast hour first. A run is treated as existing only when its final hour's file answers 200 (build_1km.py probes P048 before touching anything else).
• Fetch the last hour first. The WCS builder pulls the final forecast hour's 54 layers before the rest, so a partially-published run fails after ~54 requests, not after 1,600 (build.py).

Either way the failure mode is a clean early exit; the 15-minute schedule retries soon enough.

Append-only history as concatenated gzip members

Every published run is also appended to data/history/<slug>/<year>.jsonl.gz. The trick is how: each run is compressed as an independent gzip member and appended to the file (windgrams/publish.py). The gzip format specifies that readers process concatenated members as one stream, so any standard reader — zcat, Python's gzip — sees one JSON line per model run, in order. Existing bytes are never rewritten, which keeps git happy (appends, not churn) and makes the archive safe against interrupted writes. About 12 KB per site per run buys a permanent record of every forecast ever published — the raw material for the verification studies in where this goes next.

The ULP-parity rewrite

The derivation began life as TypeScript inside a club website. Moving it into this repository meant a Python rewrite — and a rule: the rewrite must reproduce the committed output of the original to within one double ULP (unit in the last place), verified against the previously published run as a golden file before the switch. Consumers comparing this windgram against canadarasp, or against last week's published values, must never see the implementation language.

Two details made byte-identical JSON possible:

• Integral floats print as ints. JavaScript's JSON.stringify prints 5.0 as 5; Python's json prints 5.0. The serialiser walks the profile converting integral floats to ints before dumping (publish.py), so republished files diff cleanly against their TypeScript-era ancestors.
• Dict key order matches the original serialisation — deliberate and commented in windgram.py, because a reordered key is a spurious diff in a repository whose diffs are the audit trail.

The test suite (tests/) pins the derivation with exact-value assertions, so the parity survives refactoring.

What this adds up to

A forecast product with 100% managed infrastructure: ECCC and NOAA run the models, GitHub runs the compute and the CDN, git provides the database, history, audit log, and rollback. The pipeline's only irreplaceable asset is the code and the catalogue — both of which are in front of you.