Monitoring and handling failures

Webhooks are a "quiet when working" mechanism. Precisely because of that, it is easy to miss when something is going wrong, and by the time you notice, several days of events may have slipped away. Active monitoring is essential in production.

The delivery history in the developer portal

Developer portal → App → Webhook → the relevant endpoint → the "Delivery history" tab lets you check the delivery results for the past 30 days.

Information shown

• Delivery date/time, event ID, event type
• The final status (success / retrying / dead letter)
• Each attempt's HTTP status code, response time, and response body (first 1KB)
• A "Redeliver" button (allows manual resend)

Filtering

• Status (success only / failure only / dead letter only)
• Event type
• Period (past 24 hours / 7 days / 30 days)
• Event ID / partial match of the receiving URL

Metrics to monitor

These are the minimum metrics to watch in production. Besides the developer portal's "Webhook dashboard", we recommend building them into your own monitoring infrastructure.

Metrics to hold on the receiving side

In addition to send-side monitoring, taking the following metrics on the receiving side makes it faster to identify the cause.

• Receive count (per event type)
• Signature verification failure count (an increase may mean a secret mismatch or an attack)
• Idempotency-skip count (a gauge of whether resends are increasing)
• Processing-time histogram (whether you are approaching the 10-second timeout)
• Business-processing error rate (which becomes the cause of retry loops)

Designing the alert destination

Design alerts not just so you "notice" but so you "can respond".

• Business hours: a Slack operations channel
• Nights / holidays: notify the on-call responder via PagerDuty / Opsgenie, etc.
• Minor warnings: consolidate into a daily summary email
• Dead letter occurrence: immediate notification + a weekly review

Redelivering dead letters

Events that fail all 7 automatic resends are saved as "dead letters". After fixing the cause, you can redeliver with the following procedure.

1. Select the "Dead letter" filter in the delivery history
2. Click "Redeliver" individually, or select several and "Bulk redeliver"
3. Check the results in the delivery history

Dead letters are retained for 30 days after delivery. Since they disappear after 30 days, respond before then, or re-fetch the past data via the API.

Common failure patterns and responses

1. All Webhooks are failing with 5xx

• Check whether the receiving server is down
• A recent deploy may be the cause → consider a rollback
• After recovery, "Bulk redeliver" the dead letters

2. Only some event types fail

• The processing logic for that event may have a bug
• Check the stack trace from the response body in the delivery history
• Search for the relevant event_id in the receiving-side logs

3. 401 (signature verification failure) increased

• Just after regenerating the secret, is the old secret no longer working?
• A missed environment-variable update (forgot to restart)
• Are the production / staging secrets not mixed up?

4. Frequent timeouts

• Heavy processing on the receiving side → change the design to enqueue into an internal queue
• Latency in downstream DBs or external APIs → temporarily bypass that processing
• Insufficient scale of the receiving endpoint → increase the number of instances

5. No Webhooks arrive at all

• Is the endpoint not "disabled"?
• Are the subscribed events selected correctly?
• Are you not excluded by the target-store filter?
• Is the receiving URL reachable from outside (not behind an internal VPN)?

Periodic maintenance checklist

We recommend reviewing the following about once a month.

• ☐ Dead letter count and cause over the past month
• ☐ The trend of the success rate (any degradation?)
• ☐ The trend of the P95 response time
• ☐ The secret's last-updated date (annual rotation is a good guideline)
• ☐ Deleting endpoints that are no longer needed
• ☐ Tidying up subscribed events (remove ones you don't use)

Contacting support

If you cannot identify the cause, or a ReceiptRoller-side failure is suspected, contact the developer community or the support desk. Including the following information speeds things up.

• The app ID (client ID) and the endpoint ID
• The date/time range when the issue occurred (state UTC or JST explicitly)
• Several affected event_ids
• The receiving side's response codes and a log excerpt

Related guides

• Webhook overview
• How to register a Webhook
• Signature verification and security
• Designing resends, ordering, and idempotency
• Posting a question to the community
• Back to the developer help index