Daily Build

The daily job keeps the sitemap and deploy state fresh. It is orchestrated by scripts/daily-build.sh and runs as the redecasas-daily-build.service systemd user service at 03:00 every day.

Pipeline (default: publish-only)

Skip page generation and the rotating AI refresh.
go run . generate-sitemap — always traverses public/, writes changed shards only, writes sitemap.xml last and removes stale numeric shards.
python3 scripts/validate-schema.py site/public
go run . reconcile-assets — scoped by default; refuses to upload more than 500 missing assets and honours the freeze sentinel.
Publish the sitemap shards/index/robots first.
Smart deploy of the page estate through the 500-file gate.
Write site/cache/daily-build-provenance.json and the daily success marker.

FULL_BUILD=1 opts into full page regeneration (build-pages --process-notfound). DAILY_REFRESH=1 additionally runs daily-update --force, and only has effect with FULL_BUILD=1. FORCE_REBUILD=1 implies FULL_BUILD=1.

City lists are deduplicated by URL slug before generation. IBGE returns districts/municipalities with duplicate names (225 duplicate slugs in the current cache); without deduplication each full build renders the same path twice and the final content depends on iteration order. Geo-region rows with an empty name/slug are ignored so they cannot collide with the microrregiao/{state}/index.html style index page.

Every step is fail-fast: a failed page build, sitemap validation, schema check, asset reconcile or gate/deploy aborts the run and writes site/cache/daily-build-last-failure.json. It also sends an ntfy-compatible webhook when NOTIFY_WEBHOOK_URL is configured.

The pipeline exports CREATE_PAGE_WITH_LLM=off: even a full rebuild uses cached content, not a multi-hour LLM crawl.

Freeze guard

scripts/estate-freeze.sh on --reason "..." writes site/cache/deploy/FREEZE.json and pauses the daily build, freshness monitor, the four content agents and their retry timers. When the sentinel is active:

daily-build.sh and every agent wrapper log the freeze reason and exit 0.
check-build-freshness.sh skips alerts (a paused timer is intentional).
Admin commands that trigger generation/deploy are refused.
Legacy R2 mutation paths (sync, image upload/sync, reconcile-assets, noticias folder sync) are refused.

A missing sentinel means unfrozen; a malformed sentinel is treated as frozen (fail closed). Lift with:

scripts/estate-freeze.sh off --approval https://github.com/mauricionr/brasil/issues/199#issuecomment-...

off re-enables only the daily build and freshness timers. The four content agents stay disabled until separately approved.

Deploy gate

smart-deploy writes cache/deploy/last-diff.json before uploading and refuses any deploy whose mutation count (uploads + deletions) exceeds 500. The gate is not bypassed by --yes. An over-threshold deploy requires a one-shot recorded sign-off:

go run . smart-deploy \
  --approve-large-deploy https://github.com/mauricionr/brasil/issues/199#issuecomment-... \
  --approved-by your-name \
  --summary-json cache/last-deploy.json

The approval record and JSONL audit line are written under cache/deploy/approvals/. --force --dry-run performs no uploads.

Timestamp-drift reconciliation

The 31 Oct build rewrote local pages with new article:modified_time values while R2 still held the 30 Oct bytes. reconcile-deploy-drift restores local files whose only difference is a build-volatile timestamp, with no R2 write:

cd site
go run . reconcile-deploy-drift --limit 100 --prefix casas/   # dry run
go run . reconcile-deploy-drift --write                       # apply

It refuses to run while a live incomplete deploy marker exists. Matching-hash files get their manifest mtime restored; real content drift is left untouched and reported in cache/deploy/drift-reconciliation.json.

Scheduling and installation

The tracked units live in scripts/systemd-units/:

cp scripts/systemd-units/redecasas-daily-build.{service,timer} ~/.config/systemd/user/
cp scripts/systemd-units/redecasas-boot-rebuild.service ~/.config/systemd/user/
cp scripts/systemd-units/redecasas-build-freshness.{service,timer} ~/.config/systemd/user/
systemctl --user daemon-reload
systemctl --user enable --now redecasas-daily-build.timer
systemctl --user enable --now redecasas-build-freshness.timer
systemctl --user list-timers redecasas-daily-build.timer redecasas-build-freshness.timer

The boot catch-up unit no longer sets FORCE_BUILD=1.

Skip, force and resume semantics

The build skips only when the current day’s .last-success-daily-build-YYYYMMDD marker exists.
FORCE_BUILD=1 bypasses the same-day marker; it does not trigger a full build.
FULL_BUILD=1 regenerates pages from cached content.
DAILY_REFRESH=1 (with FULL_BUILD=1) runs the rotating AI refresh.
FORCE_REBUILD=1 discards incomplete-deploy markers/journals and implies a full build.
FULL_RECONCILE=1 runs an unscoped asset audit.
If a live incomplete deploy marker exists, the next run skips page generation and resumes the deploy.

Provenance and freshness

A successful run writes site/cache/daily-build-provenance.json with the git SHA, dirty flag, start/completion timestamps, mode, full-build/daily-refresh flags, write stats, sitemap counts, schema/reconcile status and deploy summary (including gate status and diff path).

go run . build-freshness checks the provenance and sitemap locally. With --remote https://redecasas.com.br it also checks the live sitemap index, three shards and ten pages sampled across top-level prefixes.

Manual commands

# Publish-only run (default)
./scripts/daily-build.sh

# Full rebuild from cached content
FULL_BUILD=1 FORCE_BUILD=1 ./scripts/daily-build.sh

# Full rebuild + rotating AI refresh
FULL_BUILD=1 DAILY_REFRESH=1 FORCE_BUILD=1 ./scripts/daily-build.sh

# Freeze status / gate check
./scripts/estate-freeze.sh status --check

# Freshness check (local + live)
cd site && go run . build-freshness --remote https://redecasas.com.br

# Local timestamp-drift repair (no upload)
cd site && go run . reconcile-deploy-drift --write