Data pipeline

How official NDMA and PDMA figures, river levels and OpenStreetMap facilities reach Nigraan automatically, and how the jobs handle failure.

Two scheduled jobs on GitHub Actions keep Nigraan's official data and its facility map up to date, with no one pressing a button. Both write to Supabase with a service key that exists only as a GitHub Actions secret. Neither app ever holds it.

NDMA and PDMA ingest

Workflow: .github/workflows/ndma-ingest.yml Runs: three times a day, timed for after NDMA's usual publishing hours (about 13:00, 18:30 and 22:45 Pakistan time). It can also be started by hand, including as a dry run. Code: frontend/scripts/pipeline/

Each run:

1. Lists the documents each source publishes: NDMA situation reports (this season's and the previous season's archive, plus special off-season reports), NDMA advisories and relief updates, and the PDMA Sindh and PDMA KP advisories.
2. Skips documents it has already stored.
3. Downloads each new PDF and turns it into text with pdftotext (poppler).
4. Parses it. There is one parser per document layout (parseNdmaSitrep.mjs, parseNdmaRelief.mjs, parsePdmaAdvisory.mjs and others), each with tests built from real documents.
5. Places each district on the map using a gazetteer of district coordinates. A district the gazetteer does not know is reported as "not mappable", never guessed.
6. Upserts the rows into official_reports and official_relief, so running twice never duplicates anything.

When something goes wrong

• A temporary network error is retried.
• A link a source lists but does not serve (HTTP 404 or 410) is reported as a dead link, not a failure, unless every document a source lists is dead. That would mean the source moved its files.
• A real failure marks the run red and opens a GitHub issue, which closes itself on the next successful run.

River levels

River flood stages come from PMD and reservoir levels from FFC. PMD refuses requests from GitHub's servers, so that step is off in the scheduled job.

OpenStreetMap facilities

Workflow: .github/workflows/osm-facilities.yml Runs: daily. Code: frontend/scripts/buildOsmFacilities.mjs

The job asks OpenStreetMap's Overpass service for hospitals, clinics, drinking water, fire stations and ambulance stations across Pakistan. It assigns each facility's province and district from OpenStreetMap's own boundaries, and writes the result as a single row in osm_facilities.

Guards against bad data:

• Shrinkage: a run that would lose more than 10% of the facilities is refused, because a partial answer from Overpass looks like a successful one.
• Carry-forward: a known facility keeps its province and district, so most runs do not download the boundaries again.
• No empty map: if a run fails, the map keeps the last good snapshot, and a copy bundled with the site is the fallback behind that.

The public Overpass servers sometimes refuse automated requests to protect themselves. The job tries each server in turn over several rounds and never asks one again after it has refused. Most days it succeeds; on a day it does not, users see no gap.

Weather forecast

The forecast layer is not a scheduled job: each visitor's browser calls the free Open-Meteo forecast API directly, for about 55 places across Pakistan, and the site scores the results for rain, heat and wind risk.

← All technical documentation