Skill Hub — DAMA / Data

Data Lineage Working Skill

Trace data from source to consumer. Document transformations, hops, and ownership at each stage. Identify lineage gaps that create audit or trust risk.

What this skill is for

This skill helps you answer "where did this number come from?" It provides a method to trace a data element from its origin through every transformation, system, and handoff until it reaches the final consumer. The output is a documented lineage path with named owners at each hop, plus a gap list for hops that are unknown, untrusted, or unowned.

When to use this skill

A report value is questioned in an audit and no one can explain its source.
A data defect is found in a downstream system and the team does not know which upstream system introduced it.
An integration is being redesigned and the existing data flows are undocumented.
A regulation requires data lineage documentation for specific domains (e.g., financial reporting, personal data).
An AI model produces unexpected results and the training data provenance is unclear.
A system migration requires understanding which data flows must be preserved, rerouted, or retired.

Real work situations

Example 1: Audit challenge on revenue recognition

An auditor asks how the "recognized revenue" figure in the quarterly report is calculated. Finance points to a BW report. BW points to an SAP extractor. The extractor pulls from multiple tables with custom logic. No one has documented the full path. The skill traces the lineage from report column back to source transactions, documents each transformation, and identifies the owner of each step.

Example 2: Wrong customer segment in campaign

Marketing sends a campaign to the wrong customer segment. The segment value in the campaign tool does not match the segment in SAP. The skill traces the segment field from SAP through the data warehouse, a segmentation algorithm, and an API to the campaign tool. It finds that the algorithm was updated two weeks ago without notifying Marketing.

Example 3: Migration scope uncertainty

A legacy system is being retired. The team knows some data flows to SAP, but they are unsure about secondary flows to BI, a data lake, and an external partner. The skill produces a complete lineage map showing all consumers, which flows must be migrated, and which can be retired.

Example 4: AI model drift

An AI model's accuracy drops. The training data is supposed to come from a curated data product. The skill traces the data product back to its source tables and finds that a source table was modified by a recent change request, altering the feature distribution. The lineage gap was the missing change impact assessment on the AI pipeline.

Inputs required

The data element, report column, or dataset whose lineage is needed.
The consumer system or report where the data is observed.
System landscape documentation: databases, applications, integration middleware, and ETL tools.
Technical documentation: ETL jobs, API specs, database views, stored procedures, and transformation logic.
Subject matter experts who know the data flows for each system.
Access to query systems or logs to verify actual data movement (optional but recommended).

Questions to ask

What is the original system and table where this data was first recorded?
What transformations does this data undergo: aggregation, filtering, mapping, calculation, enrichment?
Which system or job moves the data from one hop to the next?
Who owns each hop: the source system, the transformation job, the integration, and the consumer?
When was the last time this lineage path was verified, and what changed since then?
What happens if a hop fails: is data delayed, lost, or stale?
Is there a parallel path where the same data flows through a different route?

Working method

Define the lineage target. Name the exact data element, report column, or dataset to trace. Include the consumer system and the business question that requires lineage.
Start from the consumer and walk backward. Identify the immediate source of the data in the consumer system. Then identify the source of that source. Continue until you reach the original creation point.
Document each hop. For each step, record: system name, component (table, file, API, job), transformation (if any), owner, and trust level (verified, inferred, claimed).
Identify transformations. For each hop where data changes (format, value, structure), document the transformation logic, the business reason, and the owner who maintains it.
Map parallel and alternate paths. Check if the same data flows through multiple routes. Document each route and note which is primary, which is backup, and which is deprecated.
Identify lineage gaps. Flag hops where: the source is unknown, the transformation logic is undocumented, the owner is missing, the path has changed without documentation, or the hop is untrusted.
Assess business risk per gap. For each gap, describe what business decision or process is affected if the lineage is wrong or unknown.
Produce the lineage map. Create a visual or tabular lineage path from source to consumer. Include hops, transformations, owners, and gap flags.
Produce the gap closure plan. For each gap, define: verification approach, documentation action, owner, and target date.
Validate with owners. Walk through the lineage map with the owner of each hop. Confirm accuracy and update where needed.

Decision rules

If a hop is claimed by an expert but not verified in system logs or code, label it "inferred" and plan verification.
If a transformation changes the business meaning of a field, document it as a semantic transformation, not just a technical conversion.
If two parallel paths produce different values for the same consumer, flag it as a data quality risk, not just a lineage variation.
If a lineage path includes a manual step (spreadsheet, email, manual upload), treat it as a high-risk hop because it is unlogged and ungoverned.
If a hop owner is missing or has left the organization, the lineage is unmaintained; assign a new owner before closing the gap.
If a source system is being retired, the lineage path is time-bound; document the retirement date and the replacement path.
If lineage is required for compliance, prioritize gaps that affect regulated data elements over non-regulated ones.

Deliverables

Data Lineage Map — source-to-consumer path with hops, transformations, and owners.
Lineage Gap Report — unknown hops, undocumented transformations, missing owners, and business risk. See Data Lineage Gap Note template.
Gap Closure Plan — actions, owners, and deadlines to close lineage gaps.

Templates

Lineage Hop Log (compact)

| Hop | System | Component | Transformation | Owner | Trust Level | Notes |
|-----|--------|-----------|--------------|-------|-------------|-------|
| 1 | SAP S/4 | Table VBRK | None | Finance | Verified | Source transaction |
| 2 | BW | Extractor 0FI_AR_4 | Filter: posting date in period | BI Team | Verified | Monthly load |
| 3 | BW | DSO ZDSO_REV | Aggregate by customer | BI Team | Verified | Sum of revenue |
| 4 | BW | Report ZREV_CUST | Format currency | BI Team | Verified | EUR conversion |
| 5 | BI Portal | Dashboard "Revenue" | None | BI Team | Inferred | Claimed by BI; not verified |

Lineage Gap Note (compact)

| Data Element | Gap | Gap Type | Business Risk | Verification Approach | Owner | Due Date |
|--------------|-----|----------|-------------|----------------------|-------|----------|
| Customer.Segment | Algorithm update not documented | Missing documentation | Wrong campaign targeting | Review algorithm version history | Data Science Lead | 2026-06-20 |
| Product.Cost | Manual spreadsheet between PLM and SAP | Untrusted hop | COGS reporting unreliable | Replace with API or document procedure | Integration Lead | 2026-07-01 |

Quality checklist

The lineage path reaches the original source system and table for the data element.
Every hop has a named system, component, and owner.
Every transformation is documented with logic and business reason.
Every hop is labeled as verified, inferred, or claimed.
All gaps are flagged with a business risk statement.
Parallel or alternate paths are documented.
The lineage map has been validated with at least one hop owner.
A gap closure plan exists with owners and target dates.

Common mistakes

Mistake: Trusting a single expert's memory without verifying in code or logs. Consequence: The lineage is wrong; the audit fails; the defect recurs.
Mistake: Documenting only the happy path and ignoring error handling, fallback loads, or manual corrections. Consequence: The lineage is incomplete; exceptions are unexplained.
Mistake: Treating lineage as a one-time documentation exercise. Consequence: The lineage is stale after the next system change; trust is lost.
Mistake: Focusing on technical hops and ignoring business ownership. Consequence: When a hop breaks, no one is accountable; resolution stalls.
Mistake: Producing a visual diagram without a tabular backup. Consequence: The diagram is hard to update; details are lost in abstraction.

Agent instructions

When using this skill, an AI agent must:

Start from the consumer. Ask the user to name the exact report, dataset, or data element that needs lineage. Do not start from "the enterprise data landscape."
Walk backward step by step. For each hop, ask: what is the immediate source? What transformation happened? Who owns it? Do not skip hops.
Label trust levels. If a hop is based on documentation, label it "from docs." If based on expert statement, label it "claimed." If verified in code or logs, label it "verified."
Flag manual steps. Any hop involving a spreadsheet, email, or manual upload is high-risk. Document it explicitly and recommend automation or formalization.
Produce artifacts, not descriptions. Output a Lineage Hop Log and a Lineage Gap Report in the templates provided. Do not write a narrative.
Link to Atlas pages. If lineage involves SAP data, link to Data Lineage or Data Quality Controls.
Handle missing information. If system access or code is unavailable, produce a discovery checklist and ask the user to verify specific hops.

Related skills

Metadata Management — when lineage gaps reveal missing metadata.
Data Quality Root Cause — when a defect needs tracing from symptom to source.
Data Governance — when lineage gaps reveal ownership problems.
Data Integration & Interoperability — when lineage crosses system boundaries.

Related Atlas pages

Data Lineage — conceptual foundation for lineage in SAP landscapes.
Data Quality Controls — controls that depend on accurate lineage.
Data Contracts — agreements at system boundaries that lineage must respect.
SAP Data Product — packaging data with lineage for consumption.

Verification status and limitations

This skill is a public working interpretation of data lineage practice. It is not official DAMA-DMBOK or ISO documentation. It has been applied in SAP, BI, and data warehouse contexts but may need adaptation for cloud-native pipelines, streaming data, or machine learning feature stores.

Limitations: This skill does not cover automated lineage extraction tools, graph-based lineage databases, or real-time lineage tracking. It focuses on manual and semi-automated lineage documentation suitable for audit, migration, and defect tracing.