Skill Hub — Work Documentation and Handover

Runbook Writing Working Skill

Write a step-by-step operational runbook that an on-call engineer can follow safely without knowing the system history.

What this skill is for

Operational incidents do not wait for the expert who knows the system best. They happen at night, on weekends, and during vacations. This skill produces an Operational Runbook: a precise, step-by-step procedure that an on-call engineer can execute safely without prior knowledge of the system history or deep domain expertise. The runbook is designed for stress: it assumes the reader is tired, under pressure, and possibly new to the environment. Every step is numbered, every decision is explicit, every command is exact, and every rollback option is stated. The output is the difference between a controlled response and a chaotic one.

When to use this skill

A critical operational procedure must be performed by on-call engineers who may not be the primary experts.
A system component fails in a predictable way and the recovery steps must be standardized to avoid ad-hoc fixes.
A routine maintenance task is performed infrequently and the team needs a reminder of the exact sequence.
An AI agent is being trained to assist with operational tasks and needs structured, unambiguous procedures.

Real work situations

SAP background job failure: runbook for queue restart

A critical background job ZIDOC_ALERT fails and the queue of failed IDocs grows. The primary engineer is on vacation. The on-call engineer must restart the job safely without causing duplicate processing or data loss. The runbook states: step 1, check SM37 for the failed job and note the job ID; step 2, check SM58 for locked entries; step 3, if no locked entries, release the job from SM37 with variant ZIDOC_ALERT_V1; step 4, monitor SM37 for 10 minutes; step 5, if the job fails again, stop and escalate to the basis team. Without the runbook, the on-call engineer restarts the job with the wrong variant, processes the same IDocs twice, and creates duplicate partner entries that require manual cleanup.

Database index rebuild: runbook for monthly maintenance

A monthly database index rebuild is required to prevent performance degradation in the customer reconciliation report. The task is performed once a month and no one remembers the exact sequence. The runbook states: step 1, check DB02 for index fragmentation above 30%; step 2, verify no long-running jobs are active in SM50; step 3, execute the rebuild script via DB13 with the scheduled variant; step 4, verify rebuild completion in DB02; step 5, run ZCUST_REP to confirm performance improvement. Without the runbook, the team skips the check for active jobs, rebuilds during peak hours, and locks the customer table for 20 minutes, blocking sales order creation.

Failover procedure: runbook for middleware switch

The primary SAP PI instance fails and the team must switch to the standby instance. The procedure involves updating RFC destinations, redirecting the message queue, and validating the first five messages. The runbook states: step 1, confirm primary instance failure via SMICM; step 2, update SM59 destination PI_PROD to point to PI_STANDBY; step 3, check SXMB_ADM for queue status on standby; step 4, release the first five messages individually and verify processing; step 5, if all five succeed, release the queue batch. If any fail, stop and call the integration architect. Without the runbook, the on-call engineer switches the destination but forgets to validate the first messages, and a mapping error on the standby instance goes undetected for two hours, corrupting 200 messages.

Inputs required

The exact procedure as performed by the expert, including every screen, transaction, and command.
The systems, environments, and clients where the procedure is executed.
The prerequisites: access rights, system state, timing constraints, and dependencies.
The success criteria: how to know the procedure worked.
The failure criteria: what to watch for, when to stop, and what to do if the procedure fails.
The rollback or recovery steps if the procedure causes unintended effects.
The escalation path: who to call if the runbook does not resolve the issue.
Historical incident data showing how the procedure has been used and what went wrong.

Questions to ask

What is the trigger for this runbook? When does an engineer know it is time to use it?
What is the first step? Is there a safety check before any action is taken?
What are the exact commands, transactions, and parameter values? Can they be copy-pasted?
What should the engineer see at each step? What output confirms the step is correct?
What could go wrong at each step? What is the stop condition?
What is the rollback plan if the procedure makes things worse?
How long should each step take? What is the timeout before escalation?
Who is the escalation contact, and what information must be provided when escalating?
Has this runbook been tested by someone who does not know the system? If not, what assumptions are unverified?

Working method

Define the trigger. State the exact condition or alert that tells the on-call engineer to open this runbook. The trigger must be unambiguous: not "when the system is slow" but "when SM50 shows more than 50% work processes in PRIV mode for more than 10 minutes."
State the prerequisites. Before any action, the engineer must verify: system availability, access rights, no conflicting jobs, and the correct environment. Include a pre-flight checklist.
Write the procedure as numbered steps. Every step must be a single action. If a step contains a decision, use a nested "if-then-else" structure. Do not combine multiple actions into one step.
Include exact commands and parameters. Use the exact transaction code, program name, command syntax, or URL. If the command has a variant, state the variant name. If a parameter must be entered, state the value or the rule for deriving it.
Include expected output for each step. State what the engineer should see after executing the step. If the output is a table, describe the expected row. If it is a message, quote the expected message text.
Document failure paths at each step. For each step, state what to do if the expected output does not appear. Include a stop condition: "If you see error X, stop and escalate. Do not proceed."
Document the rollback procedure. If the procedure modifies data or configuration, include the exact steps to undo the change. The rollback must be tested, not theoretical.
State the success criteria. Define how the engineer knows the procedure is complete and successful. Include verification steps that run after the main procedure.
Include the escalation path. Name the contact, the contact method, and the information to provide. Include a template for the escalation message.
Test the runbook. Ask an engineer who has never performed the procedure to follow the runbook in a test environment. Observe where they hesitate, where they ask questions, and where they deviate. Revise the runbook based on the test.

Decision rules

If a step has a conditional outcome, write both paths. Do not assume the engineer will know the right path.
If a command requires a password or secret, reference the secure vault. Do not embed credentials in the runbook.
If the procedure modifies production data, require a second-person review before execution. Document the review step.
If the runbook is for an emergency, keep it to one page if possible. Emergency runbooks are not comprehensive manuals; they are survival guides.
If the procedure has been updated, version the runbook and archive the old version. Do not let engineers use outdated steps.

Deliverables

Operational Runbook — Step-by-step procedure with triggers, prerequisites, exact commands, expected outputs, failure paths, rollback, and escalation. See template below.
Pre-Flight Checklist — One-page checklist of prerequisites and safety checks before the procedure begins.
Escalation Card — One-page contact list and information template for escalation.

Templates

Operational Runbook (compact)

---
artifact: Operational Runbook
id: RB-<system>-<number>
runbook_name: <Name of procedure>
severity: <P1 / P2 / P3 / routine>
environment: <Production / Test / Development>
last_tested: YYYY-MM-DD
test_result: <pass / fail / not tested>
author: <Name&gt
doc_reviewer: <Name&gt
doc_status: draft | reviewed | approved
---

## Trigger
- <Exact condition that starts this runbook>
- <Monitoring alert, ticket type, or user report>

## Prerequisites (Pre-Flight Checklist)
- [ ] System <name> is reachable
- [ ] I have access to <transaction or system>
- [ ] No conflicting jobs are running in <SM37 / SM50>
- [ ] It is within the approved maintenance window (if applicable)
- [ ] A second reviewer is available (if required for production changes)

## Procedure
1. <Action> — <Exact command or transaction>
   - Expected output: <What the engineer should see>
   - If output is not as expected: <Failure path or stop condition>
2. <Action> — <Exact command or transaction>
   - Expected output: <What the engineer should see>
   - If output is not as expected: <Failure path or stop condition>
3. Decision: if <condition>, then <step A>. Else, <step B>.
   - Step A: <Action>
   - Step B: <Action>

## Rollback
| Step | Action | Expected Output | When to Use |
|------|--------|-----------------|-------------|
| R1 | <Undo action> | <What confirms undo> | <Condition> |
| R2 | <Undo action> | <What confirms undo> | <Condition> |

## Verification
- <Step to confirm the procedure succeeded>
- <Step to confirm no side effects>

## Escalation
- Contact: <Name, role, phone / email>
- Provide: <System, incident ID, steps taken, current status, error message>
- Escalate if: <Specific conditions>

Quality checklist

The trigger is specific and measurable. An engineer knows exactly when to open the runbook.
Prerequisites are checked before any action is taken.
Every step is a single action with an exact command or transaction.
Expected output is described for every step.
Failure paths are documented at every step where failure is possible.
Rollback steps are tested and documented, not theoretical.
Success criteria are verifiable and include a post-procedure check.
Escalation contact and required information are stated.
The runbook has been tested by an engineer who did not write it.

Common mistakes

Writing a generic overview instead of a step-by-step procedure. Consequence: the engineer must interpret general advice under pressure, which leads to mistakes. "Restart the job" is not enough; specify the transaction, the job name, and the variant.
Omitting failure paths. Consequence: when a step fails, the engineer improvises. Improvisation under pressure often makes the situation worse.
Embedding passwords or secrets in the runbook. Consequence: security breach and credential rotation. Reference the secure vault or password manager.
Not testing the runbook with a new engineer. Consequence: the runbook contains assumptions that the writer did not realize. The new engineer gets stuck on step 3 and escalates unnecessarily.
Writing one runbook for multiple scenarios. Consequence: the engineer must read irrelevant sections under pressure. Split into separate runbooks for separate triggers.

Weak output vs Strong output

Weak output

A wiki page titled "IDoc Monitoring." It says: "If IDocs fail, check WE02 for errors. If the error is a mapping issue, fix the mapping in SAP PI. If the error is a data issue, correct the data in the source system. Reprocess the IDoc using BD87 or WE19. Monitor SM58 for RFC issues. Contact the integration team if the issue persists." No trigger specificity, no exact steps, no expected outputs, no failure paths, no rollback, no escalation contact details. The engineer must interpret every instruction under pressure.

Why it fails: It is an overview, not a procedure. The engineer does not know which tool to use when, what to look for, or when to stop. The runbook is not usable under pressure.

Strong output

---
artifact: Operational Runbook
id: RB-PI-2026-003
runbook_name: Restart Failed IDoc Queue for ORDERS05
severity: P2
environment: Production
last_tested: 2026-06-05
test_result: pass
author: K. Schmidt
doc_reviewer: T. Nguyen
doc_status: approved
---

## Trigger
- Monitoring alert fires for ORDERS05 IDoc queue depth > 20
- User reports sales orders not appearing in S/4HANA from EDI
- SM58 shows failed tRFC entries for destination PI_TO_S4

## Prerequisites
- [ ] S/4HANA production client 100 is reachable
- [ ] I have access to WE02, WE19, SM58, and SXMB_ADM
- [ ] No conflicting IDoc mass reprocessing job is running in SM37
- [ ] It is outside the EDI blackout window (02:00–04:00 CET)

## Procedure
1. Check WE02 for ORDERS05 IDocs with status 51 or 63 in the last 2 hours.
   - Expected output: List of failed IDocs with error text. Count should match alert.
   - If no failed IDocs are found: Stop. Check SM58 for tRFC errors instead. Escalate to basis if SM58 is clear.
2. Open the first failed IDoc. Read the error text in the status record.
   - If error text contains "mapping": Go to step 3 (mapping fix).
   - If error text contains "data" or "master data": Go to step 4 (data correction).
   - If error text contains "system" or "RFC": Go to step 5 (RFC check).
3. Mapping fix path:
   - 3a. Open SAP PI message monitoring. Find the failed message for the IDoc.
   - 3b. If the mapping error is for a known missing field (e.g., delivery block), update the mapping table ZDLV_BLOCK_MAP in SE16.
   - 3c. Reprocess the single IDoc in WE19. Verify status changes to 53 in WE02.
   - 3d. If successful, reprocess the remaining affected IDocs one by one. Do NOT use BD87 for mapping errors until the mapping is confirmed fixed.
4. Data correction path:
   - 4a. Identify the missing or incorrect master data from the error text (e.g., material not found, customer blocked).
   - 4b. Correct the data in the source system or in S/4 master data. Do NOT reprocess until the data is corrected.
   - 4c. Reprocess the corrected IDoc in WE19. Verify status 53.
   - 4d. If multiple IDocs have the same data error, fix the data once, then reprocess the batch in BD87 after verification.
5. RFC check path:
   - 5a. Check SM58 for failed tRFC entries for destination PI_TO_S4.
   - 5b. If entries are in status SYSFAIL or COMMITFAIL, check the RFC destination in SM59. Test the connection.
   - 5c. If the connection test fails, check the gateway and network status. Do NOT reprocess IDocs until the RFC destination is green.
   - 5d. Once the connection is green, execute the failed tRFC entries in SM58. Verify in WE02 that IDocs process successfully.

## Rollback
| Step | Action | Expected Output | When to Use |
|------|--------|-----------------|-------------|
| R1 | Stop reprocessing in WE19 / BD87 | No new IDocs processed | If mapping or data fix is wrong |
| R2 | Reverse master data change in the source system | Data restored to previous state | If data correction caused new errors |
| R3 | Set IDoc to status 68 (no further processing) | Status 68 in WE02 | If the IDoc must be held for manual review |

## Verification
- All failed IDocs from the initial WE02 list show status 53 or 68.
- SM58 shows no failed tRFC entries for PI_TO_S4.
- Customer service confirms that new sales orders are appearing in S/4.
- Run ZCUST_REP to verify no duplicate orders were created.

## Escalation
- Contact: K. Schmidt (Integration Architect) — +49-xxx-xxx, k.schmidt@company.com
- Provide: System (S/4 client 100), incident ID, error text from WE02, steps taken, current status, IDoc numbers affected.
- Escalate if: RFC destination is green but IDocs still fail; mapping table is updated but error persists; more than 50 IDocs are affected; or the procedure exceeds 30 minutes.

Agent instructions

AI Prompt Pattern

Role: Runbook writer for SAP operational support and on-call engineering.

Context: You have an operational procedure, an expert interview, and incident history. You need to produce an Operational Runbook that an on-call engineer can follow safely without knowing the system history.

Task: Create a structured Operational Runbook using the template below. Include a specific trigger, pre-flight checklist, exact numbered steps, expected outputs, failure paths, rollback, and escalation.

Output format: Structured Operational Runbook in Markdown with tables for rollback and escalation.

Never write a generic overview. Every step must be a single action with an exact command or transaction. The engineer should not need to interpret.
Always include a pre-flight checklist. Prerequisites must be verified before any action. Safety first.
Always include expected output for each step. The engineer must know what success looks like at each point.
Always include failure paths. If the expected output does not appear, the engineer must know what to do. Include stop conditions.
Always include rollback steps. If the procedure modifies data or configuration, the undo path must be tested and documented.
Do not embed credentials or secrets. Reference the secure vault or password manager.
Test the runbook with a new engineer. If the test reveals ambiguity, rewrite the step until it is unambiguous.
Link to Atlas diagnostics when the runbook addresses documented SAP failure modes. For example, runbooks for background jobs should reference SAP Background Job Diagnostics.

Related skills

Incident Documentation Working Skill — Use to document the incident that the runbook is designed to address.
Knowledge Article Writing Working Skill — Use to produce a quick-reference article for common symptoms; the runbook is the full procedure.
Process Documentation Working Skill — Use to document the broader process that the runbook fits into.
Incident Triage Working Skill — Use to classify and route incidents before the runbook is opened.

Related Atlas pages

SAP Background Job Diagnostics — Diagnostic context for background job runbooks.
SAP IDoc Status Diagnostics — Diagnostic context for IDoc runbooks.
SAP Interface Monitoring Diagnostics — Monitoring context for integration runbooks.

Verification status and limitations

This skill is a public working interpretation of operational runbook practices. It is not official ITIL, SAP, or DevOps documentation. It focuses on runbooks for enterprise and SAP environments where on-call engineers rotate, system history is complex, and incidents are expensive. Known limitations: it does not cover incident response coordination, communication management, or war-room facilitation. It produces the procedural document, not the response team structure. It assumes access to an expert who can verify the exact steps. It does not cover automated runbooks or cloud-native infrastructure, though the principles apply. The templates should be adapted to the organization's runbook platform or wiki.