# The storm's data (WS-214)

JSON first: the incident dashboard ([the incident page](../README.md)), Vicky's agents and the work desk read these files, not pages. Every item carries one label: **fact**, **evidence** or **hypothesis** (the meteorologist's `analysis` is a hypothesis). Times are UTC (Lisbon's own time in November).

| Path | What | Kept by | README |
|---|---|---|---|
| `signals.json` | the timeline: every row of `../timeline.md`, as data | generated by `tools/timeline.mjs` (never edit; edit `timeline.md` and run it) | below |
| `decisions/graph.json`, `decisions/sources.json` | the decision graph (deciders and their decisions) and every source | incident.researcher@web-summit.dinis | [decisions/README.md](decisions/README.md) |
| `watch/websummit.json` | Web Summit's own channels: every blog listing's posts with `date` and `modified` (J6), every sitemap's URL set (with when each URL was first seen), the organisers' last Bluesky posts; `alerts` (only grows) | generated by `tools/websummit.mjs` (incident.researcher@ runs it each wake) | below |
| `weather/*.json` | IPMA's warnings and history, communiqués, stations, the airport's METAR and TAF, IPMA's model, the analysis | incident.meteorologist@web-summit.dinis (`weather/tools/refresh.py`) | [weather/README.md](weather/README.md) |

The dashboard reads `signals.json`, `decisions/graph.json`, `watch/websummit.json` and `weather/{warnings,stations,forecast,airport,analysis}.json`. A field it uses, renamed, breaks a pane (it says which file it could not read); tell incident.dev@ in a commit (`For: incident.dev@web-summit.dinis`) and it follows.

## signals.json

`node tools/timeline.mjs incidents/2026-11-10-storm-charles`

`{ about, source, made_at, items: [ { at, approx, kind, label, key, text, sources: [{ label, url|null }] } ] }`, oldest first.

- Reads only the table rows whose first cell looks like `10 Nov, 17:42` (or `~17:42`); the paragraphs under the table are not read.
- `kind`: `signal` when the text starts "signal" (`signal:`, `signal (unofficial):`, kept as "(unofficial) …") or the source is relayed "via social.watch@"; `official` when the source names IPMA, portugal.gov.pt, websummit.com, ANEPC, S41 or the weather desk; `team` otherwise (our own record).
- `label`: `evidence` for a signal, `fact` otherwise. `key`: the row was in bold. `approx`: the time had a `~`. Only https links are kept as URLs.

## Connectors (incident.dev@web-summit.dinis)

A connector reads just the part of a source that matters, and is listed here.

| Connector | Source | Reads | Writes |
|---|---|---|---|
| `tools/timeline.mjs` | the lead's `timeline.md` | the table only | `signals.json` |
| `tools/websummit.mjs` | websummit.com: `/blog/` and every `/blog/<category>/` listing; every sitemap robots.txt names; the organisers' Bluesky (public.api.bsky.app, no login) | each listing's embedded Next.js data (`__NEXT_DATA__`): the posts' slug, title, category, `date`, `modified`, excerpt; the sitemaps' `<loc>` set only (lastmod is unreliable); the last 10 own posts (no reposts or replies) | `watch/websummit.json`; exit code 3 when a new alert was found |

## watch/websummit.json

`node tools/websummit.mjs incidents/2026-11-10-storm-charles`

`{ about, read_at, watch, sitemap: { source, count, added_this_read: [url], urls: { url: first_seen } }, blog: { source, listings: [url], posts: { url: { url, title, category, date, modified, excerpt, first_seen } } }, bluesky: { source, posts: [{ at, text, url, match }] }, alerts: [{ first_seen, channel, url, text, at?, label }], errors: [] }`

- **Why the blog (J6):** the Day 2 decision went up at 19:09 on `/blog/web-summit/important-event-update-2026/`, which no sitemap lists; we learned of it 43 minutes later, from the press. Every blog listing is now read, and a category named only by a post (e.g. `partners`) is read too. A blog alert is `new` (a post not seen before), `modified` (its `modified` changed) or, on the first read only, `recent` (published or modified in the last 36 hours). The WordPress REST API answers 404 on this site (a Next.js front end), so the listings' embedded data is the one read.
- **Every 2 minutes** (the researcher's ask, 21:03): `tools/websummit-poll.sh <repo checkout> <scratch dir>` runs a copy of `websummit.mjs` **pinned when the poll starts** (its sha256 printed) with `--quick` (`/blog/` only), against a scratch copy of the watch file. The loop runs no git command. On an alert it stops; incident.dev@'s session is woken, reads the alert as data and commits the watch file itself, with a fixed subject and the site's words only in the body, quoted. The poll lives as long as that session (it stops within 2 hours and restarts each wake). Redesigned at 21:30 after security.repos@'s MEDIUM (report 2026-11-10T2130Z): the first version reset to `dev` every loop and ran what it found there, and put site text in commit subjects. The full read (every listing, the sitemaps, Bluesky) runs on the agents' wakes.
- **robots.txt is obeyed:** it is read on every run; a disallowed path (today only `/api/`) is never fetched (`robots_disallow` in the file).
- **Two dates per post, each named:** `modified` is the listing's `Post.modified` (`modified_field`, read at `read_at`); `page` holds the post page's own JSON-LD `date_published` and `date_modified` with their `read_at`, for posts of the last 48 hours. They can differ: "Important event update" reads modified 19:58 in the listing and 19:25 in its page's JSON-LD (21:11).
- **Listings disagree, and the newest wins:** at 21:33 `/blog/` said a post was modified at 20:06 while `/blog/web-summit/` still said 19:58 (a cached copy). `modified` is the newest value any listing gives, `modified_by_listing` keeps each listing's value, and an older value never moves it back (no false "modified" alert).
- **The post's own words:** for posts of the last 48 hours, `page.text` (its `__NEXT_DATA__` `post.content`, tags stripped, at most 4,000 characters) and `page.text_sha256`. A `text_changed` alert fires when the hash changes, whatever the dates say. (At 21:32 `modified` moved and we could not tell what had changed; from 21:40 we can.)
- **Sitemap URLs:** every URL ever seen is kept; a URL alerts once, the first time it appears, whatever its words (the set changes between reads).
- An **alert** is a new blog post or a changed one (above), a new sitemap URL or an own post whose text matches `watch` (since 20:04 also Thursday/quinta, day 2 and 3, PITCH, tickets, refunds, talks, speakers, streams, side events: the open items after the Wednesday decision). It is a fact that the organiser published it; **a match is not a decision**: read it.
- The first read records the URL set and the newest post and raises no alert; after that only a new URL, or a post newer than the last read's newest, can raise one. `alerts` keeps every alert ever raised, with `first_seen`.
- The dashboard also reads Bluesky **live in the browser** every minute (its public API allows any origin), so a post shows on the page without waiting for this file, CI or Pages' cache. The sitemap has no such header, so it needs this connector.

The weather connectors are the meteorologist's own (`weather/tools/refresh.py`). The researcher's parser asks (`decisions/sources.json`, `parser_ask`) are next.