> ## Documentation Index
> Fetch the complete documentation index at: https://docs.arcbeam.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Data Lineage

> Trace AI outputs back to source documents

**Data lineage** is the ability to trace an AI-generated output back through the retrieved documents to the original source files. It answers the question: "Why did the AI say that?"

<Note>
  Data lineage creates a complete audit trail from user question → AI response → retrieved documents → original source files
</Note>

## What Is Data Lineage?

<CardGroup cols={2}>
  <Card title="What data was used" icon="database">
    See exactly which documents and information the AI accessed to generate its response
  </Card>

  <Card title="When it was created or updated" icon="clock">
    Track timestamps to identify outdated information that might affect accuracy
  </Card>

  <Card title="Where it came from" icon="map-pin">
    Trace back to the original source files like PDFs, documents, or web pages
  </Card>

  <Card title="How it influenced the output" icon="chart-line">
    Understand the connection between retrieved data and the AI's final answer
  </Card>
</CardGroup>

## Why Data Lineage Matters

### Debug Bad Outputs

When an AI gives a wrong answer, data lineage helps you trace the problem to its source and fix it systematically.

<Steps>
  <Step title="View the trace">
    Find the problematic interaction in your traces to see what happened during that specific AI interaction
  </Step>

  <Step title="Check retrieved documents">
    See what documents the AI used to generate its answer - these are the documents pulled from your vector database
  </Step>

  <Step title="Verify document content">
    Check if the documents contain incorrect information that led to the wrong answer
  </Step>

  <Step title="Trace to source">
    Find the original source file (PDF, document, web page) where the incorrect information came from
  </Step>

  <Step title="Update the source">
    Correct the information in the original file to fix the root cause
  </Step>

  <Step title="Re-sync database">
    Update your vector database with the corrected content so the AI can access accurate information
  </Step>

  <Step title="Verify improvement">
    Test to confirm the AI now gives correct answers using the updated information
  </Step>
</Steps>

<Warning>
  Without lineage, you'd be guessing what caused the problem and might fix the wrong thing.
</Warning>

### Demonstrate Compliance

For regulated industries like healthcare, finance, and legal, data lineage provides the audit trail needed to meet regulatory requirements.

<CardGroup cols={2}>
  <Card title="Auditability" icon="scale-balanced">
    Prove which data influenced each decision with a complete record of data sources used for every AI output
  </Card>

  <Card title="Transparency" icon="eye">
    Show regulators the full data trail from question to answer to source documents
  </Card>

  <Card title="Accountability" icon="user-check">
    Identify who updated source documents and when to maintain responsibility for data quality
  </Card>

  <Card title="Reproducibility" icon="rotate">
    Re-run queries and verify consistent results to demonstrate reliable AI behavior
  </Card>
</CardGroup>

### Measure Data Quality

Data lineage lets you correlate data characteristics with output quality to understand what makes your AI perform better.

<CardGroup cols={2}>
  <Card title="Document Freshness" icon="clock">
    Do recent documents lead to better answers? Track how document age affects AI accuracy
  </Card>

  <Card title="Source Reliability" icon="star">
    Do certain sources have higher accuracy? Identify your most trustworthy data sources
  </Card>

  <Card title="Document Length Impact" icon="ruler">
    Does document length affect relevance? Understand optimal chunking sizes
  </Card>

  <Card title="Metadata Importance" icon="tags">
    Which metadata fields matter most? Discover what additional context improves results
  </Card>
</CardGroup>

### Prioritize Maintenance

Know which documents need updates by seeing how they're actually being used in your AI system.

<CardGroup cols={3}>
  <Card title="Update First" icon="arrow-up">
    High-traffic, outdated docs - These are used frequently but contain old information
  </Card>

  <Card title="Review Content" icon="magnifying-glass">
    Frequently retrieved, low-satisfaction docs - Users see these often but aren't happy with results
  </Card>

  <Card title="Consider Removing" icon="trash">
    Never-retrieved docs - These aren't helping anyone and add noise to your system
  </Card>
</CardGroup>

## Compliance & Governance

Use lineage for regulatory requirements in industries like healthcare, finance, and legal where proving data provenance is mandatory.

<CardGroup cols={2}>
  <Card title="Audit Trails" icon="scroll">
    Complete records of data usage
  </Card>

  <Card title="Data Retention" icon="hourglass">
    Document lifecycle tracking
  </Card>
</CardGroup>

### Audit Trails

Generate comprehensive reports showing the complete data trail for any AI decision.

**What audit reports include:**

* Which specific data influenced which decisions (document IDs and sources linked to outputs)
* When data was accessed (timestamps for every retrieval event)
* Who updated source documents (author and modification history)
* Full chain of custody (from source file creation through AI usage)

<Check>
  These reports can be exported and submitted to regulators to demonstrate compliance with data governance requirements.
</Check>

### Data Retention

Track the complete lifecycle of documents to ensure compliance with retention policies.

**What you can track:**

* When documents were added to your system (initial embedding date)
* How long they've been in use (age of current version)
* When they should be reviewed or removed (based on retention rules)
* Compliance with retention policies (automated alerts for documents approaching limits)

<Warning>
  Some regulations require removing data after a certain period. Lineage helps you identify which documents need deletion and verify complete removal.
</Warning>

## Best Practices

Follow these practices to get the most value from data lineage and maintain data quality over time.

### Maintain Source Attribution

Always include source metadata when embedding documents into your vector database. This metadata is what makes lineage tracking possible.

<CardGroup cols={2}>
  <Card title="Good: Complete metadata" icon="circle-check">
    Include source filename, page number, update date, and version number when embedding each document chunk
  </Card>

  <Card title="Bad: Missing metadata" icon="circle-xmark">
    Embedding documents without any metadata means you can't trace outputs back to sources
  </Card>
</CardGroup>

**Example of good metadata:**

* Source: "refund\_policy\_2024.pdf"
* Page: 3
* Updated: "2024-12-01"
* Version: "v2.1"

### Use Consistent Naming

Keep source names consistent across updates so you can track changes to the same document over time.

| Approach   | Example                                                   | Result                                            |
| ---------- | --------------------------------------------------------- | ------------------------------------------------- |
| **Good** ✅ | "refund\_policy\_2024.pdf" → "refund\_policy\_2025.pdf"   | Easy to track the evolution of your refund policy |
| **Bad** ❌  | "refund\_policy\_2024.pdf" → "new\_policy\_final\_v2.pdf" | Can't tell these are the same document            |

<Tip>
  Use a naming convention like `[topic]_[year].pdf` or `[topic]_v[version].pdf` to make tracking easier.
</Tip>

### Track Update History

Log when sources are updated so you can correlate changes with quality improvements or regressions.

**What to track:**

* Set `updated_at` timestamp whenever you modify a source file
* Increment version numbers with each update (v1.0 → v1.1 → v2.0)
* Keep a changelog of what changed in each version

<Info>
  When you see quality drop after a certain date, you can check which documents were updated around that time.
</Info>

### Review Lineage Regularly

Schedule regular reviews (weekly or monthly) to proactively maintain your data quality.

**What to review:**

* Check which sources are used most frequently in your AI interactions
* Verify high-traffic sources contain current, accurate information
* Update outdated but frequently-used documents as a priority
* Remove or archive documents that haven't been retrieved in months

<Check>
  Regular reviews prevent the slow degradation of AI quality that happens when source documents become gradually outdated.
</Check>

### Document Your Sources

Maintain a source catalog (like a spreadsheet or database) that tracks important information about each source file.

| What to Document              | Why It Matters                                                            |
| ----------------------------- | ------------------------------------------------------------------------- |
| **What each source contains** | Quickly understand which source to update when information changes        |
| **Who owns it**               | Know who to contact for questions or updates                              |
| **Update frequency**          | Set expectations for how often it should be reviewed                      |
| **How to update it**          | Process for making changes (especially for generated or imported sources) |

<Tip>
  Include a link to the original source location (SharePoint, Google Drive, etc.) so you can quickly find and update it.
</Tip>

## Common Use Cases

Real-world scenarios where data lineage solves critical problems in AI systems.

<CardGroup cols={2}>
  <Card title="Root Cause Analysis" icon="magnifying-glass-chart">
    Debug incorrect AI outputs
  </Card>

  <Card title="Content Gap Identification" icon="circle-question">
    Find missing information
  </Card>

  <Card title="Quality Correlation" icon="chart-line">
    Improve output quality
  </Card>

  <Card title="Compliance Reporting" icon="file-contract">
    Prove data provenance
  </Card>
</CardGroup>

### Root Cause Analysis

**Problem:** Your AI gives a wrong answer and you need to find and fix the cause.

<Steps>
  <Step title="View the trace">
    Look at the specific interaction where the AI gave the wrong answer to see all retrieved documents
  </Step>

  <Step title="Check document content">
    Read through the retrieved documents to find the outdated or incorrect information
  </Step>

  <Step title="Trace to source">
    Follow the lineage to identify which PDF or source file contained the bad information
  </Step>

  <Step title="Update and re-embed">
    Fix the source file, then re-chunk and re-embed the corrected content
  </Step>

  <Step title="Verify the fix">
    Check new traces with similar questions to confirm the AI now gives correct answers
  </Step>
</Steps>

### Content Gap Identification

**Problem:** Your AI can't answer certain types of questions because it lacks the necessary information.

<Steps>
  <Step title="Filter low-relevance traces">
    Find traces where retrieved documents had low relevance scores (meaning no good matches were found)
  </Step>

  <Step title="Review query patterns">
    Look at which user questions resulted in poor document matches to identify patterns
  </Step>

  <Step title="Identify missing topics">
    Determine what subject areas or information types are missing from your knowledge base
  </Step>

  <Step title="Create new sources">
    Write or acquire documents covering the missing topics
  </Step>

  <Step title="Embed and monitor">
    Add the new documents to your vector database and track improvement in answer quality
  </Step>
</Steps>

### Quality Correlation

**Problem:** Some documents lead to better AI outputs than others, but you don't know which ones or why.

<Steps>
  <Step title="Correlate usage with feedback">
    Connect document retrieval events with user feedback scores to see patterns
  </Step>

  <Step title="Identify high-quality sources">
    Find which sources consistently lead to high user satisfaction and correct answers
  </Step>

  <Step title="Identify low-quality sources">
    Find which sources are frequently retrieved but lead to poor feedback or incorrect outputs
  </Step>

  <Step title="Improve or remove bad sources">
    Either fix the low-quality sources or remove them entirely from your system
  </Step>

  <Step title="Expand good source coverage">
    Create more documents similar to your high-quality sources to improve overall system quality
  </Step>
</Steps>

### Compliance Reporting

**Problem:** Regulators or auditors need proof of which data influenced AI decisions.

<Steps>
  <Step title="Generate time-bound report">
    Create a report for the specific time period being audited (e.g., Q4 2024)
  </Step>

  <Step title="Show source access">
    List all source files that were accessed during AI interactions in that period
  </Step>

  <Step title="Demonstrate audit trail">
    Provide the complete chain from user questions to AI responses to retrieved documents to source files
  </Step>

  <Step title="Export for submission">
    Export the lineage data in a format suitable for regulatory submission (PDF, CSV, etc.)
  </Step>
</Steps>

## Next Steps

<CardGroup cols={2}>
  <Card title="Datasets & Data Sources" icon="database" href="/v0/core-concepts/datasets-and-data-sources">
    Connect vector databases to enable lineage
  </Card>

  <Card title="Trace Issues to Source Data" icon="link" href="/v0/debugging/trace-issues-to-source-data">
    Debug problems using lineage
  </Card>

  <Card title="Track Data Changes" icon="clock" href="/v0/data-insights/track-data-changes">
    Monitor impact of source updates
  </Card>

  <Card title="Add Data Sources" icon="plug" href="/v0/setup/add-data-sources">
    Set up your first dataset
  </Card>
</CardGroup>