Test Evidence Review Working Skill

What this skill is for

Test execution produces evidence: screenshots, logs, IDoc numbers, order numbers, and status records. This evidence is the proof that the system was tested and that it behaved as expected. However, evidence is often incomplete, inconsistent, or untraceable. Screenshots are missing. Test cases are marked "passed" without evidence. Expected results are not stated. Defects are linked to the wrong requirement. This skill reviews the test evidence systematically to ensure it is complete, accurate, traceable, and credible. The output is a Test Evidence Review Report that identifies gaps, requests corrections, and certifies the evidence package for QA sign-off and audit.

When to use this skill

• Test execution has finished and the evidence must be reviewed before QA sign-off.
• An audit or compliance review requires evidence that testing was performed and documented.
• Test cases are marked passed but the evidence is missing, vague, or inconsistent.
• A defect was found during testing and the evidence does not clearly show the failure and the retest.
• The project lead suspects that test results were padded or that failed cases were marked passed without proper resolution.
• Test evidence must be handed over to operations or support as part of the release documentation.

Real work situations

Missing screenshots in a regulatory audit

A pharmaceutical company requires documented evidence for every validated SAP change. The test team executed 50 cases and marked them all passed. The evidence review discovers that 12 cases have no screenshots, 5 screenshots show a different system client than the test environment, and 3 cases have no traceability to requirements. The review must request the missing evidence, verify the client consistency, and add the requirement links before the audit. Without the review, the audit fails and the release is blocked for two weeks.

Inconsistent IDoc status evidence

A test case verifies that customer IDocs process successfully within five minutes. The evidence shows the IDoc number and a status 53 screenshot, but the timestamp on the screenshot is from a different day than the test execution date. The review must flag the inconsistency, request the correct evidence, and verify the actual processing time. Without the review, the release proceeds with unverified performance claims, and the IDoc processing time degrades in production.

Defect retest without before-and-after evidence

A defect in the pricing procedure was fixed and retested. The test case is marked passed, but the evidence only shows the after-fix screenshot. The review requires the before-fix evidence (the failure) and the after-fix evidence (the success) to prove that the fix addressed the reported issue. Without the review, the defect could have been closed on a different scenario, and the original issue recurs in production.

Inputs required

• Test Case Document with planned cases, expected results, and traceability links.
• Test execution log or report showing which cases were executed, by whom, and when.
• Test evidence attachments: screenshots, logs, documents, IDoc numbers, order numbers, and report outputs.
• Defect log showing defects found, their status, and retest evidence.
• Traceability matrix showing requirements mapped to test cases and defects.
• Environment configuration records showing which system and client were used for each test.
• Review criteria from the QA policy or regulatory standard defining what evidence is required.

Questions to ask

• Does every executed test case have evidence attached, and does the evidence match the case description?
• Does the evidence show the system identifier (client, URL, environment) used for the test?
• Does the evidence include the expected result, the actual result, and a clear indication of pass or fail?
• Are dates and timestamps in the evidence consistent with the test execution schedule?
• For failed cases, is there evidence of the failure, the defect report, and the retest after fix?
• Does every case trace to a requirement, and does every defect trace to a case and a requirement?
• Is the evidence readable, complete, and stored in a location that is accessible for audit?
• Are there cases that were marked passed but the evidence is missing, ambiguous, or from a different test?

Working method

1. Gather the evidence package. Collect all test execution reports, evidence attachments, defect logs, and traceability matrices. Ensure the package is complete and versioned.
2. Verify evidence completeness. For every test case marked as executed, check that evidence is present. If a case is missing evidence, flag it as "evidence gap" and request the tester to provide it.
3. Verify evidence accuracy. Check that the evidence matches the test case description. For example, if the case tests a credit block, the evidence must show the blocked order, not a successful order. If the case tests IDoc processing, the evidence must show the IDoc number and status. Mismatches are flagged as "evidence mismatch."
4. Verify environment consistency. Check that the system, client, URL, or environment name visible in the evidence matches the approved test environment. Evidence from production or a different client is invalid and must be rejected.
5. Verify traceability. Confirm that every test case links to a requirement and that every defect links to a test case and a requirement. Orphan evidence (cases with no requirement, defects with no case) is flagged as "traceability gap."
6. Verify defect closure evidence. For every defect marked closed, confirm that there is evidence of the original failure and evidence of the retest after fix. Defects closed without both pieces of evidence are flagged as "closure incomplete."
7. Verify timestamp consistency. Check that dates and timestamps in the evidence are consistent with the test execution schedule. Evidence dated before the test started or after it ended is suspicious and must be investigated.
8. Document findings in the review report. Use the template below. For each finding, state the case ID, the finding type (gap, mismatch, traceability, closure, timestamp), the severity, and the required correction.
9. Request corrections. Send the report to the test lead or tester with a deadline for corrections. Re-review the corrected evidence before finalizing the report.
10. Certify the evidence package. Once all findings are resolved, produce a final Test Evidence Review Report that certifies the evidence as complete, accurate, and traceable. Include the reviewer name, date, and scope of the certification.

Decision rules

• If a test case has no evidence, it is not considered executed. Mark it as "not executed" until evidence is provided.
• If evidence is from a different environment than the approved test system, reject it and request re-execution in the correct environment.
• If a defect is closed without retest evidence, reopen the defect and require retest evidence before closing it again.
• If evidence is ambiguous but the case is clearly correct, request clearer evidence rather than accepting ambiguity.
• If a timestamp inconsistency is minor (e.g., screenshot taken one minute after case completion due to file save delay), accept it with a note. If it is major (e.g., different day), investigate and reject.
• If the traceability matrix is missing or incomplete, require it to be updated before the evidence is certified.
• If all findings are resolved and the evidence package is complete, the package is certified for QA sign-off and audit.

Deliverables

• Test Evidence Review Report — Per test cycle or per release. Contains findings, corrections requested, and certification status. See template below.
• Corrected Evidence Package — Updated evidence attachments that address all findings.
• Certification Statement — Formal statement that the evidence package is complete, accurate, and traceable, signed by the reviewer.

Templates

Test Evidence Review Report (compact)

---
artifact: Test Evidence Review Report
cycle: <Test cycle or release>
reviewer: <Name>
date: <YYYY-MM-DD>
---

## Scope
- Test cases reviewed: <count>
- Evidence items reviewed: <count>
- Defects reviewed: <count>
- Requirements traced: <count>

## Findings
| Case ID | Finding Type | Severity | Description | Required Correction | Status |
|---------|--------------|----------|-------------|---------------------|--------|
| TC-001 | Evidence gap | High | No screenshot attached | Provide screenshot of saved order | Open |
| TC-002 | Mismatch | Medium | Screenshot shows different material than case specifies | Re-execute with correct material or update case | Open |
| TC-003 | Traceability | High | Case not linked to any requirement | Link to REQ-003 and update matrix | Open |
| TC-004 | Closure | High | Defect DEF-001 closed without retest evidence | Provide retest screenshot and re-close | Open |
| TC-005 | Timestamp | Low | Screenshot timestamp 2 minutes after execution | Accept with note | Closed |

## Summary
- Findings open: <count>
- Findings closed: <count>
- Evidence package status: Not certified / Certified pending corrections / Certified

## Certification
- **Status:** Not certified / Certified
- **Scope:** Evidence package for cycle <name>
- **Certified by:** ___________________ Date: _________

Quality checklist

• Every executed test case has evidence that is present, readable, and matches the case description.
• Evidence is from the approved test environment, not from production or an unapproved system.
• Every case traces to a requirement, and every defect traces to a case and a requirement.
• Every closed defect has both failure evidence and retest evidence.
• Timestamps in evidence are consistent with the test execution schedule.
• The review report documents every finding with a type, severity, and required correction.
• All findings are resolved before the evidence package is certified.
• The certification statement is signed, dated, and scoped to a specific test cycle or release.

Common mistakes

• Accepting test results without reviewing evidence. Consequence: cases are marked passed with no proof. The audit fails, the release is blocked, and the QA sign-off is questioned.
• Ignoring environment mismatches in evidence. Consequence: evidence from production or a different client is accepted as valid. The test does not actually prove the change works in the target environment.
• Allowing defect closure without retest evidence. Consequence: defects are assumed fixed but were never verified. They recur in production, causing rework and trust erosion.
• Reviewing evidence only for failed cases. Consequence: passed cases with missing or incorrect evidence are overlooked, and the pass rate is inflated. The coverage claim is false.
• Storing evidence in inaccessible locations. Consequence: when the audit or support team needs the evidence, it cannot be found. The certification is meaningless if the package is lost.

Weak output vs Strong output

Weak output

A simple confirmation: "All test cases have been executed and passed. Evidence is attached in the shared folder. No issues found." No review of individual evidence items, no traceability check, no environment verification, no defect closure verification, no findings, no certification.

Why it fails: It is not a review; it is a statement of trust. If evidence is missing, mismatched, or from the wrong environment, the confirmation is false. It cannot be used in an audit or a retrospective. It provides no basis for improvement.

Strong output

---
artifact: Test Evidence Review Report
cycle: UAT Wave 2
reviewer: D. Kharlanau
date: 2026-06-12
---

## Scope
- Test cases reviewed: 45
- Evidence items reviewed: 45
- Defects reviewed: 5
- Requirements traced: 12

## Findings
| Case ID | Finding Type | Severity | Description | Required Correction | Status |
|---------|--------------|----------|-------------|---------------------|--------|
| TC-001 | Evidence gap | High | No screenshot of saved order | Provide screenshot | Open |
| TC-002 | Mismatch | Medium | Screenshot shows material M-002 instead of M-001 | Re-execute with M-001 | Open |
| TC-003 | Traceability | High | Not linked to requirement | Link to REQ-003 | Open |
| DEF-001 | Closure | High | Closed without retest evidence | Provide retest screenshot | Open |
| TC-005 | Timestamp | Low | 2 min after execution | Accept with note | Closed |

## Summary
- Findings open: 4
- Findings closed: 1
- Status: Certified pending corrections

## Certification
- **Status:** Certified pending corrections
- **Scope:** UAT Wave 2 evidence package
- **Certified by:** D. Kharlanau, 2026-06-12

Agent instructions

AI Prompt Pattern

Role: Test evidence reviewer for an enterprise SAP project.

Context: You have test execution reports, evidence attachments, defect logs, and a traceability matrix. You need to produce a Test Evidence Review Report that verifies completeness, accuracy, traceability, and environment consistency.

Task: Review every executed case for evidence presence and accuracy. Verify environment consistency. Verify traceability. Verify defect closure evidence. Document findings, request corrections, and certify the package when resolved.

Output format: Test Evidence Review Report in Markdown, using the compact template with a findings table and a certification section.

• Never certify an evidence package without reviewing every executed case. A sampling approach is acceptable only if the sampling method is documented and approved; otherwise, review every case.
• Always reject evidence from the wrong environment. Evidence from production or a different client does not prove the change works in the target environment.
• Always require retest evidence for closed defects. A defect is not closed until the original failure and the post-fix success are both documented.
• Always verify traceability. Orphan cases and orphan defects are gaps that must be fixed before certification.
• Always document findings with severity and required corrections. Vague findings like "check evidence" are not actionable.
• Do not invent evidence, timestamps, or traceability links. Use the inputs provided. If inputs are missing, flag the gap as a finding.
• Link to Atlas diagnostics when findings touch SAP-specific evidence. For example, IDoc status evidence should be evaluated against SAP IDoc Status Diagnostics for correct status codes.

Related skills

• Test Case Design — Provides the cases and expected results that evidence must match.
• QA Review and Sign-Off — Uses the certified evidence package for the quality gate.
• Defect Triage and Classification — Provides the defect log that the evidence review verifies.
• Traceability Matrix — Provides the requirement-to-case mapping that the review verifies.

Related Atlas pages

• SAP IDoc Status Diagnostics — Reference for evaluating IDoc evidence and status codes.
• SAP Background Job Diagnostics — Context for reviewing batch job execution evidence.
• SAP Credit Management Diagnostics — Reference for credit block evidence verification.

Verification status and limitations

This skill is a public working interpretation of test evidence review practices. It is not official ISTQB, GxP, or SAP documentation. It focuses on enterprise contexts where test evidence must be credible for audit, compliance, and release decisions.

Known limitations: the skill assumes that evidence is stored in a retrievable format (screenshots, logs, documents). It does not cover video recordings or automated test tool exports, which may require specialized review methods. It does not address regulatory validation standards (e.g., FDA 21 CFR Part 11) that require electronic signatures and audit trails on the evidence itself. The review process is manual; large-scale automated testing may require sampling strategies not covered here.