Reliability and recovery

Version 0.1.1 adds recovery and operational safeguards. Passing automated tests is not an independent security audit, an uptime guarantee, or proof that a particular deployment's backups are enabled.

Database migrations and release order#

Hosted request handlers do not migrate the database. The same rule applies to any production process and any cloud database connection, including a developer shell outside Vercel. Only local development/test databases initialize automatically. A cold function checks the schema version and reuses one initialized app. A missing/outdated schema returns JSON 503 with a request ID. This avoids repeating catalog imports and schema writes across cold instances when database capacity is limited.

1. Verify a complete encrypted backup of the existing database and files.

2. Run isolated tests against local fixtures, never against production resources.

3. Load the selected production environment explicitly and run the additive migration before deploying code that requires it:

   hljs shCopy

   node --env-file=.env.production.local scripts/migrate.mjs
   

4. Deploy, then check /api/health, a signed-out feed, a signed-in profile, password recovery, and private attachment access.

5. Keep preview credentials separate. Do not run fixture seeding or local data resets with production credentials.

A database marker newer than this code is rejected rather than downgraded. The migration command accepts either no arguments (full migration/finalization) or one recognized preparation action; typos and combined actions fail before opening the database. For the additive Workspaces v6/v7 rollout, follow the 0.2 migration sequence rather than prematurely finalizing the marker. Prepared v6 readiness requires the immutable-update trigger and pagination/feedback indexes as well as the tables, feedback anchor and completion record.

The version marker is written only after all migrations complete. Existing module migrations remain idempotent so an interrupted migration can be retried. Run one migration operator at a time. The committed Vercel region is dub1, matching the original deployment's Ireland database; independent deployments should choose a region near their own primary database.

Encrypted backups#

npm run backup -- create exports a consistent read transaction to a standalone SQLite database and copies every object referenced by its uploads table. Objects are immutable by ID. Missing objects and size mismatches fail the job instead of producing a misleading successful backup.

The archive is compressed and encrypted with AES-256-GCM, a fresh random nonce, an authentication tag, and SHA-256 file checksums inside the encrypted manifest. Account data and password verifiers are retained. Sessions, saved-device sessions, OAuth states/link challenges, pending registrations, password reset codes, and incomplete upload reservations are excluded. Restoring a backup requires users to sign in again; old authentication challenges cannot be resurrected.

Every creation command decrypts and restores its archive into a new temporary directory, checks SQLite integrity/foreign keys and table counts, verifies every file's checksum and metadata, then removes that temporary copy. A failed verification prevents artifact publication.

Activate scheduled backups#

The Encrypted backup and restore check GitHub Actions workflow runs daily, with a manual workflow_dispatch option. It needs these repository Actions secrets:

• TURSO_DATABASE_URL, TURSO_AUTH_TOKEN
• S3_ENDPOINT, S3_REGION, S3_BUCKET, S3_ACCESS_KEY_ID, S3_SECRET_ACCESS_KEY
• BACKUP_ENCRYPTION_KEY: a random 64-character hexadecimal value, generated locally with a cryptographic RNG. Store a separate offline/password-manager copy. Losing the key makes recovery impossible.

Use a dedicated database backup token with read/write permission only if running the included maintenance step; without maintenance the exporter itself only reads the database. Keep access restricted to this application's database and storage project.

Only the .ddb encrypted artifact is uploaded to GitHub, separate from the application storage provider, with seven-day retention. No plaintext exports or user data belong in Git. The default input budget is 50 MiB per backup, intended to leave headroom for seven copies under a small artifact-storage allowance. It is deliberately lower than the application's upload cap. If data grows beyond it, the workflow fails visibly: review retention, artifact storage allowance and an independent backup destination before increasing BACKUP_MAX_BYTES. Do not disable file inclusion to make a failing full backup appear successful. Existing account usage and provider limits must also be reviewed.

Scheduling depends on GitHub Actions being enabled and available; monitor successful runs and artifact existence. A committed workflow is not an activated or verified production backup. Planned recovery point is one successful daily snapshot; recovery time depends on dataset size and operator restoration checks. Keep an additional downloaded encrypted copy if you need recovery after GitHub account loss.

Verify or recover an archive#

With the correct key available only in the process environment:

npm run backup -- verify backups/example.ddb
npm run backup -- restore backups/example.ddb /absolute/path/to/new-recovery-directory

Restore refuses an existing target directory. It never overwrites a running database or cloud bucket. For recovery of the hosted service, first import the restored SQLite into a new libSQL database, upload the restored uploads/ files with the same object IDs to a new private bucket under uploads/<id>, run current migrations against the new database, test access with a separate preview, and only then switch production environment settings. Recheck file authorization and revoke/rotate credentials if recovering from compromise. Provider import procedures vary; do not blindly execute unreviewed SQL against production.

Error visibility and availability#

API errors and initialization failures log structured events with a generated request ID, error category, and source locations. They do not log passwords, bodies, cookies, email addresses, SQL or arbitrary dependency error messages. Requests over two seconds produce slow_request events. Vercel log retention remains provider-controlled; there is no permanent external log store in this release.

/api/health makes a minimal database query. The Production health workflow checks it about twice hourly, retries transient failures, and fails the workflow if the API is unhealthy. Enable GitHub Actions failure notifications for the maintainer account and run it manually once after deployment. This is periodic detection, not real-time alerting or an end-to-end OAuth/mail/storage availability test.

When database capacity is exceeded, callers receive JSON 503 with Retry-After; failed app initialization is held for 15 seconds before another attempt. The frontend handles plain-text/HTML hosting errors, bounds request time, does not automatically replay mutations, and shows a support request reference when available.

Quotas and cleanup#

Defaults, configurable through server-only environment variables:

Mail attempts are reserved before provider calls, including failed deliveries. Review provider timezones/limits and usage outside Devdevian; the application cannot budget unrelated sends on a shared provider account. Upload reservations are committed before object writes, so parallel instances cannot oversubscribe the configured byte budget. A failed object cleanup retains its reservation for later maintenance.

npm run maintenance deletes expired authentication records, old quota counters, and objects left by incomplete uploads older than one hour. It does not delete published uploads or community content. The backup workflow runs maintenance before its snapshot.

These limits protect specific paid/quota-sensitive operations. They are not global DDoS protection. Configure hosting firewall/bot rules and spending limits appropriate to your provider. The synchronous libSQL adapter still limits per-instance throughput during remote I/O; isolated local load results must not be presented as hosted capacity guarantees.

Regression checks#

npm test covers recovery-code expiry/replay/attempt caps, session revocation, backup tampering/wrong keys/missing objects, restoration, schema markers, quota atomicity, API errors, existing private-space boundaries, and ordinary forum behavior. The libSQL/S3 fixture matrix exercises the cloud adapter contract. npm run test:browser covers desktop, tablet, mobile navigation, dialogs, media and recovery UI.

npm run test:load uses a disposable local database with 500 posts and 1,500 replies and sends 210 requests with concurrency 10. It cannot contact production database or storage. Measure actual hosted latency separately with a small read-only smoke check; do not stress the live free-tier service or create fake public activity.

Workspaces release-candidate rehearsal#

tests/workspaces-rc.test.mjs checks future-schema refusal, preparation flag mistakes, damaged readiness and non-Vercel production startup without DDL. Its encrypted archive round trip includes project associations, immutable multi-file snapshots, original feedback/replies and a private attachment. It boots a restored prepared-v6 copy, requires fresh login, verifies private access/revocation and simulates legacy writes in the pre-finalization rollback window before re-preparing/finalizing. This is a simulation of old-writer SQL behavior, not execution of an old production binary or a hosted database restore. The source fixture stays unchanged.

The Workspace browser runner commits a request but drops its response: the saved uncertain outcome survives reload and no POST is replayed automatically. Definitive quota refusal remains editable. CLI creation also records intent before sending and stops if intent cannot be saved. A known published post ID is reused for linking; neither client silently merges or retries an uncertain mutation.

The local load fixture now includes 499 project work requests (50 private), 100 immutable revisions, 1,000 remarks and 1,500 replies. It checks private aggregate exclusion, bounded pages, indexed request pagination and 300 reads at concurrency 10. These are regression budgets for isolated fixtures, not production capacity or availability guarantees.