Exit to App
How It Works

Failure Handling & Recovery

Crash-safety, database recovery, worker restart handling, and reconciliation protocols.

Financial systems must guarantee consistent state even across sudden power failure, container restarts, or RPC outages. MOI implements multi-tier recovery:

  • Orphaned Execution Recovery: On worker startup, recoverOrphanedExecutions() identifies any records stuck in SUBMITTED state, queries Celo RPC for transaction receipts, and advances them to CONFIRMED or FAILED.
  • Automated Database Snapshots: PostgreSQL automated WAL archiving and daily checksummed backups verified by SHA-256 integrity hashes.
  • Deterministic State Resumption: Idempotency keys on all webhook ingestion endpoints ensure network retries never trigger duplicate execution records.