Best practices for production operation

Split into three environments

Separate secrets, databases, and URLs completely.

Monitoring (minimum)

• API call success rate: alert below 99%
• Rate-limit hit rate: alert at 1% or higher
• Webhook receive success rate: alert below 99%
• Token refresh failures: immediate notification
• Signature verification failures: alert if they occur continuously
• Processing-queue backlog: alert when it exceeds a threshold

Deployment

• Automated tests in CI: type checks, unit tests, API contract tests
• Canary: roll out a new version to a subset of traffic first
• Rollback plan: revert to the previous version with a single command
• Migration strategy: release DB schema changes in two steps (add → remove)
• Feature flags: deploy new features OFF, and switch them ON in production

Following ReceiptRoller-side versions

• Subscribe to "Announcements"
• Log the Sunset / Deprecation headers and get notified
• Review your use of deprecated APIs in a monthly review
• When a new API version is released, verify it in staging first

The basic structure of incident response

1. Detection: a monitoring alert fires
2. First move: the on-call responder starts within 15 minutes
3. Identify the scope of impact: what is broken for whom
4. Temporary mitigation: rollback, feature disable, manual handling
5. Communication: notify the affected stores / users
6. Permanent fix: fix the root cause
7. Retrospective: a postmortem (improvement, not blame)

Keep documentation in order

• Runbook: response steps for common incidents
• Architecture diagram: the connection points with ReceiptRoller
• Contacts: ReceiptRoller support desk, internal team contacts
• List of environment variables: what value, and where it comes from
• Recovery steps: DB restore, deploy rollback, secret regeneration

Test accounts and test data

• For E2E tests that involve production access, use a dedicated test store and test user
• Make test credit card numbers and test email addresses constants
• As a rule, do not test with production data (risk of handling personal information)

Related guides

• How to think about data protection
• Access control and audit logs
• SLA, uptime, and the checking procedure during an incident
• Back to the developer help index