Integration Error Handling Working Skill

What this skill is for

This skill helps you classify integration failures by type, design a retry policy that handles transient failures without causing storms, define dead letter criteria and reprocess workflows, create escalation paths for unresolvable errors, and document the error handling policy so operators and consumers know what to expect.

When to use this skill

• You are designing a new integration and need to define what happens when it fails.
• Recurring failures are exhausting support capacity with manual retries.
• A dead letter queue is filling up and no one knows the reprocess procedure.
• Retry logic is causing cascading failures or thundering herds on recovering systems.
• An audit finding notes missing message reliability or traceability.
• Consumers report they cannot tell whether a failure is retryable or fatal.

Real work situations

Situation 1: Manual IDoc retries 20 times per day

Customer master IDocs fail in SAP with status 51 due to a mapping validation error. The AMS operator manually reprocesses each IDoc in BD87 after fixing the data. This happens 20 times per day. There is no automated retry, no root cause fix, and no escalation when the volume spikes.

Situation 2: API consumer does not know whether to retry

A downstream system calls a REST API and receives a 500 error. The consumer has no documentation on whether this is transient or permanent. It retries immediately, then again, then again. The API is already overloaded; the retries make it worse. Eventually the consumer gives up and drops the request.

Situation 3: Dead letter queue fills with no review process

An event broker routes failed events to a dead letter topic. The topic grows by hundreds of messages per week. There is no scheduled review, no reprocess runbook, and no owner. After six months, the team discovers that a whole category of orders was never processed.

Situation 4: Retry storm on recovering downstream system

A downstream billing system goes down for maintenance. When it comes back, all queued messages retry simultaneously. The billing system is overwhelmed and fails again. The cycle repeats until the queue is manually paused.

Inputs required

• Interface inventory with failure history.
• Middleware retry capabilities: max retries, backoff options, dead letter support.
• SAP error handling tools: AIF, IDoc status, qRFC, tRFC, BD87, SM58.
• SLA requirements: max acceptable delay, ordering requirements.
• Dead letter queue or topic configuration.
• Consumer retry behavior: what consumers do on failure.
• Escalation contacts: who handles unresolvable errors per interface.
• Business impact per failure type: cost of delay, cost of data loss.

Questions to ask

• Is this failure transient (network, timeout, downstream restart) or permanent (schema mismatch, validation error, bad data)?
• What is the maximum retry count, and what happens after the last retry?
• What is the backoff strategy: fixed, linear, exponential, or custom?
• Who reviews dead letter messages, and how often?
• How is a poison message identified and quarantined?
• What is the business impact of delayed reprocessing versus immediate rejection?
• How does the consumer know a message failed, and what action should it take?
• If message ordering matters, how do retries preserve or recover sequence?

Working method

1. Classify failure types per interface. For each interface, list the failure modes and classify each as transient, permanent, or data-related. Document the symptom and root cause pattern.
2. Define retry policy per type. For transient failures, define max retries, backoff strategy, and jitter. For permanent failures, define immediate dead letter or rejection.
3. Define dead letter criteria. Specify when a message goes to dead letter: max retries exceeded, permanent failure, poison message, schema validation failure.
4. Design escalation path. For dead letter messages, define: who reviews, how often, what tools they use, and when to escalate to development or business.
5. Create reprocess runbook. Document the safe steps to reprocess a dead letter message: validation, environment, order preservation, idempotency check, confirmation.
6. Test failure scenarios. Simulate each failure type in a controlled environment. Verify retry behavior, dead letter routing, alert firing, and runbook accuracy.
7. Monitor retry and dead letter metrics. Track retry rate, dead letter growth rate, time-to-review, and reprocess success rate. Alert on anomalies.
8. Review weekly. For the first month after go-live, review failure patterns, tune retry parameters, and update runbooks based on real behavior.

Decision rules

• If the failure is a network error or timeout, retry with exponential backoff and jitter.
• If the failure is a schema mismatch or data validation error, send to dead letter immediately; do not retry.
• If retry count exceeds three without success, escalate to human review and stop automatic retry.
• If the dead letter queue grows by more than ten messages per day, trigger a process review.
• If the consumer is external, provide explicit error codes distinguishing retryable from non-retryable failures.
• If SAP IDoc fails, use AIF for structured error handling, categorization, and reprocessing.
• If message ordering matters, retry in sequence or use idempotency so out-of-order reprocessing is safe.
• If a downstream system is recovering, use circuit breaker or rate-limited retry to prevent thundering herds.
• If a message fails with the same permanent error on every retry, classify it as a poison message and quarantine it.

Deliverables

• Error Handling Policy per Interface — Failure types, retry policy, dead letter criteria. See template below.
• Retry Configuration Specification — Max retries, backoff, jitter, circuit breaker settings.
• Dead Letter Process — Review schedule, owner, tools, escalation.
• Escalation Matrix — Who handles what, when to escalate, contact paths.
• Reprocess Runbook — Safe steps to reprocess dead letter messages.

Templates

Error Handling Policy

---
artifact: Error Handling Policy
id: EHP-001
interface: Interface ID
status: draft | reviewed | approved
---

## Interface


## Failure type classification

| Failure Type | Symptom | Root Cause | Retryable | Max Retries | Backoff | Action After Max |
|--------------|---------|------------|-----------|-------------|---------|------------------|
| Transient timeout | HTTP 504, connection reset | Network, downstream restart | Yes | 3 | Exponential 1s→8s | Dead letter + alert |
| Schema mismatch | HTTP 400, validation error | Contract change, bad payload | No | 0 | N/A | Dead letter + alert |
| Auth failure | HTTP 401, 403 | Expired token, permission change | Yes* | 2 | Fixed 5s | Dead letter + alert |
| Rate limit | HTTP 429 | Consumer over limit | Yes | 5 | Exponential 1s→30s | Dead letter + alert |
| Data validation | IDoc status 51, AIF error | Mapping, missing field | No | 0 | N/A | Dead letter + business review |
| Downstream unavailable | HTTP 503, queue full | Maintenance, overload | Yes | 3 | Exponential 2s→16s | Dead letter + alert |

*Auth failure is retryable only if token refresh is automated.

## Dead letter configuration

| Attribute | Value |
|-----------|-------|
| Dead letter destination | Topic / Queue / Table |
| Retention | 30 days |
| Review frequency | Daily |
| Review owner | Name / Team |
| Escalation threshold | > 10 messages / day |

## Reprocess runbook reference


## Consumer error contract
| Error Code | Retryable | Consumer Action |
|------------|-----------|-----------------|
| E001 | Yes | Backoff 5s and retry |
| E002 | No | Log and stop; contact support |

## Owner


## Review date

Quality checklist

• Every interface has classified failure types with clear retryable / non-retryable distinction.
• Retry policy is documented with max count, backoff, and jitter.
• Dead letter process exists with named reviewer and review frequency.
• Escalation path is defined with thresholds.
• Reprocess runbook is tested in a non-production environment.
• No silent message loss: every failure is logged, alerted, or routed to dead letter.
• Consumer receives actionable error information (retryable flag, error code, description).
• Circuit breaker or rate limit is configured for downstream recovery scenarios.

Common mistakes

• Mistake: Infinite retry on permanent failures. Consequence: Resources are wasted, logs fill, and the real problem is never escalated.
• Mistake: No dead letter process. Consequence: Failed messages are dropped or accumulate indefinitely.
• Mistake: Silent failures (no alert, no log, no dead letter). Consequence: Data loss is discovered days or weeks later during reconciliation.
• Mistake: Retry without idempotency, causing duplicates. Consequence: Downstream systems process the same message multiple times.
• Mistake: Different error handling for the same interface depending on which consumer reports it. Consequence: Inconsistent behavior and untraceable failures.

Agent instructions

• Classify failures before designing retry: Do not assume all failures are transient. Separate network, auth, schema, data, and downstream failures.
• Use SAP-specific tools: For SAP IDoc and RFC failures, reference AIF, BD87, SM58, and qRFC monitoring. Link to Atlas diagnostics.
• Ensure idempotency: Every retryable interface must have an idempotency mechanism documented.
• Produce artifacts: Generate an Error Handling Policy, reprocess runbook, and escalation matrix. Do not stop at general advice.
• Avoid generic language: Do not write "implement robust error handling." Write "Retry transient timeouts up to 3 times with exponential backoff; route schema mismatches to dead letter immediately."
• Handle missing information: If middleware dead letter capabilities are unknown, list them as a blocking dependency.
• Link to Atlas diagnostics: Reference Retry and Error Handling, Dead Letter Queue, Idempotency, and SAP Integration Error Handling Diagnostics.

Related skills

• Integration Observability — Detect failures so error handling can trigger.
• API Integration — Define error contracts for APIs.
• Event-Driven Architecture — Handle event delivery failures.
• Interface Ownership — Assign dead letter review and escalation owners.

Related Atlas pages

• Retry and Error Handling — Conceptual foundation.
• Dead Letter Queue — Dead letter patterns.
• Idempotency — Safe retry design.
• SAP Integration Error Handling Diagnostics — SAP-specific patterns.
• SAP qRFC / tRFC Diagnostics — Queue retry and monitoring.
• SAP IDoc Status Diagnostics — IDoc failure classification.

Verification status and limitations

This skill is a public working interpretation of integration error handling practice. It is not official SAP, AWS, Azure, or vendor documentation. SAP-specific guidance references standard AIF, IDoc, and RFC mechanisms available in common releases; custom implementations may differ. The skill assumes middleware provides some form of retry and dead letter capability; if not, the first step is capability assessment, not policy design.