Catalog operations

Open AI agent: production collection and publication use the original repository YouthOpp/data-pipeline.

Only the original YouthOpp/data-pipeline default branch collects on merge, manual dispatch, and at minute 17 every six hours (UTC); development forks do not publish production catalogs. GitHub may delay or disable inactive scheduled workflows; inspect Actions rather than treating the cron expression as proof of execution. Collection is serialized, fixtures run before network access, and all-source failure refuses publication.

Each successful collection publishes catalog.json, collection-report.json, contributors.json and manifest.json under catalog-<run-id>-<attempt>. The manifest records the immutable release tag and SHA256/byte size of all three data files. catalog-latest is a mutable pointer: its manifest is uploaded last, after the complete immutable release exists. State restoration captures that manifest; website builds use the committed catalog-release.json pointer to pin the manifest digest and release tag, then download catalog.json and contributors.json from that release and verify both assets before building. Before the first website pointer exists, the build uses catalog-latest. Do not independently download multiple latest assets and assume a consistent snapshot.

Prior-state restoration also captures the pointer manifest and verifies the immutable catalog. The first upgrade from older releases verifies GitHub's asset digest; missing digest, corruption, unexpected schema or network/permission errors stop publication. Only an actual HTTP404 for the absent catalog-latest release seeds an empty first state.

After successful pointer publication, the workflow preserves the newest 30 published versioned catalog releases, catalog-latest, and the current run's snapshot. It removes only tags matching catalog-<digits>-<digits>, never unrelated releases, drafts or prereleases. This bounds snapshot storage; opportunity records remain preserved across bounded feeds and source failures. Retention errors fail the run after data publication, so inspect the release pointer before retrying. Run attempts have different tags.

Recovery: inspect the collection report and failed step, fix reviewed source/configuration errors, then rerun or manually dispatch the default branch. Never publish an empty catalog to conceal an outage. A prior valid immutable manifest can be restored to catalog-latest to roll back producer state; for the website, also run the version notification with that manifest to update catalog-release.json and trigger the rollback build. Retain the matching files and verify SHA256. The upgraded website requires contributors.json; rollback must use a compatible post-upgrade snapshot or jointly restore the earlier consumer. Older catalog snapshots remain supported for producer state restoration. Frontend Pages configuration is independent of successful data releases.

Contributor collection runs in this producer after opportunity collection and before manifest creation. Repository fetch targets are runtime configuration; published contributor project references use canonical YouthOpp names, while history_repositories preserves the actual collection provenance in the data. Partial history or API identity mapping failures are explicitly recorded. If every history fetch fails, publication stops and the prior successful release pointer remains. PR validation imports scoring fixtures without fetching remote history. Older manifests remain valid for restoring prior catalog state; new publication requires contributor integrity metadata.

Website refresh after publication

After immutable assets and the latest manifest are published, scripts/notify-website.mjs updates catalog-release.json in the original website main using a short-lived GitHub App installation token passed as the WEBSITE_REPO_TOKEN environment variable; no personal token secret is required. Cloudflare's existing Git integration builds the commit. The pointer pins the immutable release and manifest digest; datasets remain in pipeline Releases. Collection or publication failures cannot update the pointer. Missing or rejected credentials fail notification without rolling back the release. Repeating the same notification does not create another commit. See Cloudflare setup for token scope, build settings and acceptance checks.