> ## Documentation Index
> Fetch the complete documentation index at: https://docs.arcbeam.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Document Usage Analytics (Coming Soon)

> Track how individual documents are used across all traces

Document usage analytics show you which pieces of content in your knowledge base are actually being used by your AI—and which are sitting unused.

## Viewing Document Usage

<Steps>
  <Step title="Navigate to your datasets">
    Go to **Data → Datasets**
  </Step>

  <Step title="Select a dataset">
    Choose the dataset you want to analyze
  </Step>

  <Step title="View document list">
    Browse the list of documents in your dataset
  </Step>

  <Step title="Access detailed usage">
    Click on any document to see comprehensive usage metrics and analytics
  </Step>
</Steps>

## Key Metrics per Document

Understanding these metrics helps you identify which documents are performing well and which need attention.

### Retrieval Count

**How many times this document was retrieved** across all traces.

| Count Range     | What It Means                           | What To Do                                                  |
| --------------- | --------------------------------------- | ----------------------------------------------------------- |
| High (100+)     | Core document that's frequently needed  | Keep this up to date—it's critical to your AI's performance |
| Medium (10-100) | Regularly used and important            | Monitor for accuracy and keep content fresh                 |
| Low (1-10)      | Occasionally useful for niche topics    | Verify the content is still needed and relevant             |
| Zero            | Never retrieved, potentially irrelevant | Consider removing or improving embeddings                   |

### Unique Queries

**How many different queries** retrieved this document.

<Info>
  A document with 100 retrievals from only 2 unique queries is less valuable than one with 100 retrievals from 50 unique queries.
</Info>

| Uniqueness Level | What It Means              | Use Case                                                      |
| ---------------- | -------------------------- | ------------------------------------------------------------- |
| High Uniqueness  | Broadly applicable content | Answers many different questions and serves diverse use cases |
| Low Uniqueness   | Narrow use case            | Same questions repeatedly, limited applicability              |

**Real-world comparison:**

| Document   | Retrievals | Unique Queries | Assessment                               |
| ---------- | ---------- | -------------- | ---------------------------------------- |
| Document A | 200        | 80             | Versatile content serving many use cases |
| Document B | 200        | 5              | Narrow focus, repetitive queries         |

### Average Relevance Score

**Mean relevance score** when this document was retrieved.

| Score Range | Meaning         | What It Tells You                             |
| ----------- | --------------- | --------------------------------------------- |
| > 0.8       | Strong match    | Highly relevant, excellent semantic alignment |
| 0.6 - 0.8   | Good match      | Relevant and useful for the queries           |
| 0.4 - 0.6   | Weak match      | Marginally relevant, may not be helpful       |
| \< 0.4      | Very weak match | Likely not helpful, poor semantic match       |

<Warning>
  **Red Flag Alert**: High retrieval count + low average relevance = your retrieval system is finding this document, but it's not actually a good match.

  **Consider these fixes:**

  * Improve the document content to be more focused
  * Fix or regenerate embeddings
  * Adjust retrieval parameters or similarity thresholds
</Warning>

### Last Retrieved

**When this document was most recently used**.

| Time Range           | Status            | What It Indicates                       |
| -------------------- | ----------------- | --------------------------------------- |
| Recent (\< 7 days)   | Actively used     | Currently in use for ongoing operations |
| Moderate (7-30 days) | Regularly used    | Still relevant and useful               |
| Old (30-90 days)     | Infrequently used | Worth reviewing for continued relevance |
| Very Old (> 90 days) | Rarely/never used | Possibly outdated or no longer relevant |

### User Feedback Correlation

**How users rated traces that used this document**.

This metric shows you the quality of your documents from your users' perspective:

<CardGroup cols={3}>
  <Card title="Thumbs Up Count" icon="thumbs-up" color="#16a34a">
    Traces using this document that received positive feedback
  </Card>

  <Card title="Thumbs Down Count" icon="thumbs-down" color="#dc2626">
    Traces using this document that received negative feedback
  </Card>

  <Card title="Satisfaction Rate" icon="percentage" color="#2563eb">
    Percentage of positive feedback overall
  </Card>
</CardGroup>

| Feedback Pattern | What It Might Mean                                                                                                      | Recommended Action                                                                                                    |
| ---------------- | ----------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------- |
| High Thumbs Down | Document contains wrong/outdated information, is unclear or confusing, or doesn't answer the questions users are asking | Review and update immediately                                                                                         |
| High Thumbs Up   | Document is valuable and helpful to users                                                                               | Keep it well-maintained, consider creating similar high-quality content, and use it as a template for other documents |

## Document Details Page

Click on any document to access comprehensive information and insights.

### Full Content

The complete text of this document chunk, exactly as stored in your vector database. This lets you see precisely what your AI has access to when this document is retrieved.

### Metadata

All fields synced from your vector database provide context about the document:

* **Source** — Original file or URL where this content came from
* **Last updated** — When the source was last modified
* **Tags** — Categories or labels for organization
* **Custom fields** — Any other metadata you've configured

### Usage Timeline

A graph showing document retrievals over time reveals important patterns:

| Pattern Type        | What It Looks Like              | What To Investigate                                              |
| ------------------- | ------------------------------- | ---------------------------------------------------------------- |
| Trending Documents  | Sudden increase in usage        | Something made this document more relevant—find out what changed |
| Declining Documents | Used to be popular, now ignored | Investigate why usage dropped—may be outdated or replaced        |
| Seasonal Patterns   | Spikes at certain times         | Plan updates around predictable usage patterns                   |

### Recent Traces

List of most recent traces that retrieved this document:

* **Trace ID** — Link to the full trace for detailed analysis
* **Query** — What the user asked
* **Timestamp** — When this retrieval happened
* **User feedback** — Thumbs up/down if available

<Tip>
  Click through to see the full trace and understand exactly how this document was used in context.
</Tip>

### Related Documents

Documents that are frequently retrieved alongside this one reveal important relationships:

| Insight Type      | What It Reveals                                  | Action To Take                                        |
| ----------------- | ------------------------------------------------ | ----------------------------------------------------- |
| Document Clusters | Related topics that are often retrieved together | Ensure these documents are well-maintained as a group |
| Coverage Gaps     | Missing related documents that users need        | Add new content to fill the gaps                      |
| Redundancy Check  | Multiple docs with similar content               | Consolidate duplicates for clarity                    |

## Sorting and Filtering Documents

Find exactly the documents you need to review with powerful sorting and filtering options.

### Sort Options

| Sort By           | What It Shows You                            | When To Use                             |
| ----------------- | -------------------------------------------- | --------------------------------------- |
| Most Retrieved    | Your core documents that power your AI       | Finding high-impact content to maintain |
| Least Retrieved   | Unused content that may need attention       | Identifying candidates for removal      |
| Highest Relevance | Best semantic matches across retrievals      | Finding well-matched, quality documents |
| Lowest Relevance  | Poorly matched documents needing improvement | Identifying documents to fix or remove  |
| Most Recent       | Recently added or updated documents          | Tracking new content performance        |
| Oldest            | Long-standing content that may need refresh  | Finding potentially outdated documents  |

### Filter Options

| Filter Type           | Example Use                              | What It Helps You Find                     |
| --------------------- | ---------------------------------------- | ------------------------------------------ |
| Retrieval Count Range | Documents with 10-100 retrievals         | Medium-usage documents for review          |
| Relevance Threshold   | Only docs with >0.7 average relevance    | High-quality, well-matched documents       |
| Date Range            | Documents retrieved in last 30 days      | Recently active content                    |
| User Feedback         | Filter by positive or negative feedback  | Documents linked to user satisfaction      |
| Source Filter         | Filter by original file or URL           | Content from specific sources              |
| Custom Metadata       | Filter using your custom metadata fields | Documents with specific tags or attributes |

## Use Cases

Here's how to apply document usage analytics to solve real problems.

### Find High-Impact Documents to Update

**Goal**: Focus your limited time on updating what matters most.

<Steps>
  <Step title="Sort by Most Retrieved">
    Identify your most-used documents
  </Step>

  <Step title="Check last updated dates">
    Review the "Last Updated" date for your top 20 documents
  </Step>

  <Step title="Identify outdated high-impact docs">
    Find old documents with high retrieval counts
  </Step>

  <Step title="Update source files">
    Make improvements to those documents in your original source files
  </Step>

  <Step title="Re-sync data source">
    Sync the updates to your vector database
  </Step>
</Steps>

<Check>
  **Result**: Maximum impact from your update efforts—you're improving the documents that matter most.
</Check>

### Remove Unused Documents

**Goal**: Clean up your vector database for better performance.

<Steps>
  <Step title="Filter for zero retrievals">
    Find documents with **zero retrievals** in the last 60 days
  </Step>

  <Step title="Review each document">
    Examine the content and metadata
  </Step>

  <Step title="Determine the root cause">
    Ask: "Is this truly irrelevant, or just poorly embedded?"
  </Step>

  <Step title="Remove irrelevant docs">
    Delete truly irrelevant documents from your vector database
  </Step>

  <Step title="Re-embed valuable docs">
    For documents that should be findable, regenerate embeddings
  </Step>
</Steps>

<Check>
  **Result**: Leaner database, faster retrievals, and better-quality results.
</Check>

### Investigate Low-Quality Documents

**Goal**: Fix documents that are retrieved but not helpful.

<Steps>
  <Step title="Filter for problematic docs">
    Find documents with **high retrieval count** + **low average relevance** (\< 0.6)
  </Step>

  <Step title="Read the document content">
    Review what's actually in the document
  </Step>

  <Step title="Check recent traces">
    Look at traces that recently used this document
  </Step>

  <Step title="Diagnose the issue">
    Determine if the problem is:

    * **Content is wrong/outdated** → Update the content
    * **Content is fine but embeddings are bad** → Re-embed the document
    * **Retrieval parameters are too loose** → Adjust similarity threshold
  </Step>

  <Step title="Apply the fix">
    Make the necessary changes based on your diagnosis
  </Step>
</Steps>

<Check>
  **Result**: Better retrieval quality and more relevant documents in your results.
</Check>

### Track Content Gaps

**Goal**: Find topics where you need more documentation.

<Steps>
  <Step title="Find traces with poor retrieval">
    Look at traces with no relevant documents retrieved
  </Step>

  <Step title="Group by topic">
    Organize by topic or query type to find patterns
  </Step>

  <Step title="Identify patterns">
    Example: "All pricing questions have no good docs"
  </Step>

  <Step title="Create new content">
    Write new documentation to fill the identified gaps
  </Step>

  <Step title="Verify improvement">
    Add to vector database and confirm retrieval improves
  </Step>
</Steps>

<Check>
  **Result**: Comprehensive knowledge base with fewer "I don't know" responses.
</Check>

### Measure Update Impact

**Goal**: Verify that updating a document improved performance.

<Steps>
  <Step title="Record baseline metrics">
    Note retrieval metrics before update (count, relevance, feedback)
  </Step>

  <Step title="Update the document">
    Make improvements in your source files
  </Step>

  <Step title="Re-sync to Arcbeam">
    Push the updated document to your vector database
  </Step>

  <Step title="Wait for data collection">
    Allow 1-2 weeks for meaningful data
  </Step>

  <Step title="Check new metrics">
    Review the same metrics again
  </Step>

  <Step title="Compare before/after">
    Analyze the impact of your changes
  </Step>
</Steps>

<Check>
  **Result**: Data-driven confirmation that your updates actually helped.
</Check>

### Debug Wrong Answers

**Goal**: Fix AI responses that provide incorrect information.

<Steps>
  <Step title="View the trace">
    When AI gives a wrong answer, view the trace in Arcbeam
  </Step>

  <Step title="Check retrieved documents">
    See which documents were used to generate the response
  </Step>

  <Step title="Review document content">
    Click through to see the full content of each document
  </Step>

  <Step title="Diagnose the issue">
    Determine if:

    * Documents contain incorrect information
    * Correct documents weren't retrieved
    * Wrong documents were prioritized
  </Step>

  <Step title="Fix the root cause">
    * If documents are wrong, update the source files
    * If documents are missing, add new content
    * If retrieval is broken, adjust parameters
  </Step>

  <Step title="Re-sync and verify">
    Update your dataset and confirm improved outputs
  </Step>
</Steps>

<Check>
  **Result**: Trace errors back to source data and fix them systematically.
</Check>

### Compliance and Audit Trail

**Goal**: Demonstrate data provenance for compliance requirements.

<Steps>
  <Step title="Generate trace report">
    Create a report for the required time period
  </Step>

  <Step title="Export document usage">
    Download data showing which documents were accessed
  </Step>

  <Step title="Trace to original files">
    Use source attribution to link back to original files
  </Step>

  <Step title="Document the audit trail">
    Show complete chain from AI output to source document
  </Step>

  <Step title="Record access logs">
    Document which users accessed which information
  </Step>
</Steps>

<Check>
  **Result**: Full provenance chain for regulatory compliance and auditing.
</Check>

## Exporting Document Usage Data

Export document metrics for external analysis and reporting.

<Steps>
  <Step title="Navigate to datasets">
    Go to **Data → Datasets**
  </Step>

  <Step title="Select your dataset">
    Choose the dataset you want to export
  </Step>

  <Step title="Click Export">
    Click **Export Usage Data**
  </Step>

  <Step title="Choose format">
    Select CSV or JSON format
  </Step>

  <Step title="Download">
    Save the file for analysis
  </Step>
</Steps>

### CSV Export Contents

The exported CSV includes all key metrics:

* Document ID
* Content (truncated or full based on your selection)
* Retrieval count
* Unique queries
* Average relevance score
* Last retrieved date
* User feedback statistics

### Common Use Cases for Exports

| Use Case             | What You Can Do                                            | Who Benefits                                 |
| -------------------- | ---------------------------------------------------------- | -------------------------------------------- |
| Custom Dashboards    | Create visualizations in Excel, Tableau, or other BI tools | Data analysts and leadership teams           |
| Team Collaboration   | Share metrics with content teams and stakeholders          | Content managers, product teams, and editors |
| Archival & Reporting | Maintain historical records and generate periodic reports  | Compliance teams and auditors                |
| Trend Analysis       | Track changes in document performance over time            | Data scientists and ML engineers             |
| Content Planning     | Identify gaps and prioritize content creation efforts      | Documentation teams and content strategists  |

## Setting Up Alerts

Get notified about important changes to stay proactive.

### Critical Document Alerts

Monitor your most important documents for issues:

<Steps>
  <Step title="Tag critical documents">
    Mark your essential documents with a "critical" tag
  </Step>

  <Step title="Set retrieval drop alerts">
    Get notified if retrieval count drops suddenly
  </Step>

  <Step title="Monitor relevance scores">
    Alert when relevance score declines below threshold
  </Step>

  <Step title="Track negative feedback">
    Receive alerts on negative user feedback
  </Step>
</Steps>

<Info>
  **Example Alert**: "Pricing Policy Doc" retrieval count dropped 80% in the last week → investigate potential retrieval issue or content problem.
</Info>

### New Content Alerts

Track newly added documents to ensure they're working properly:

<Steps>
  <Step title="Enable new document alerts">
    Get notified when a new document is synced
  </Step>

  <Step title="Monitor initial usage">
    Track its first 30 days of usage automatically
  </Step>

  <Step title="Verify retrieval">
    Confirm it's being retrieved as expected
  </Step>
</Steps>

## Best Practices

Follow these practices to get the most value from document usage analytics.

### Review Top 20 Documents Monthly

| Review Task        | What To Check                                               | Why It Matters                                   |
| ------------------ | ----------------------------------------------------------- | ------------------------------------------------ |
| Check Accuracy     | Verify your most-used documents contain correct information | Errors in frequently-used docs impact many users |
| Verify Freshness   | Confirm documents are up to date with current information   | Outdated content leads to wrong AI responses     |
| Review Feedback    | Read user feedback on traces that used these documents      | User feedback reveals quality issues             |
| Update When Needed | Make improvements based on your findings                    | Continuous improvement maintains quality         |

### Investigate Zero-Retrieval Documents

Every month, check documents that are never retrieved:

| Question To Ask            | What To Look For                                | Possible Action                 |
| -------------------------- | ----------------------------------------------- | ------------------------------- |
| Are they truly irrelevant? | Does this content apply to your AI's use cases? | Remove if genuinely not needed  |
| Are embeddings broken?     | Is the content findable with semantic search?   | Regenerate embeddings if broken |
| Should they be removed?    | Is this taking up space without value?          | Delete to optimize database     |

### Correlate with User Feedback

When users give thumbs down to AI responses:

<Steps>
  <Step title="View the trace">
    Open the trace with negative feedback
  </Step>

  <Step title="Check retrieved documents">
    See which documents were used in that response
  </Step>

  <Step title="Look for patterns">
    If the same document appears in many negative traces, you've found the problem
  </Step>

  <Step title="Update immediately">
    Fix that document as soon as possible
  </Step>
</Steps>

### Track Seasonal Patterns

<Info>
  Some documents may have seasonal usage patterns that affect your update planning:

  * **Tax documents** — Retrieved heavily in April
  * **Holiday policy docs** — Spike in November-December
  * **Budget documents** — Active at fiscal year end

  Plan your content updates and reviews around these patterns.
</Info>

### Use Related Documents Feature

When creating new content:

| Task               | What To Do                                                 | Benefit                               |
| ------------------ | ---------------------------------------------------------- | ------------------------------------- |
| Check Related Docs | Review related documents for similar topics                | Understand existing content landscape |
| Avoid Duplication  | Ensure new content doesn't duplicate existing docs         | Prevent confusion and redundancy      |
| Fill Gaps          | Create content that bridges gaps between related documents | Provide comprehensive coverage        |

## Next Steps

<CardGroup cols={2}>
  <Card title="Dataset Analytics" icon="chart-bar" href="/v0/data-insights/dataset-analytics">
    Understand patterns across entire datasets
  </Card>

  <Card title="Track Data Changes" icon="clock-rotate-left" href="/v0/data-insights/track-data-changes">
    Monitor how your data evolves over time
  </Card>

  <Card title="Find Problematic Traces" icon="magnifying-glass" href="/v0/debugging/find-problematic-traces">
    Identify and debug issues in your AI responses
  </Card>

  <Card title="Understand Retrieval Quality" icon="sparkles" href="/v0/debugging/understand-retrieval-quality">
    Deep dive into retrieval performance metrics
  </Card>
</CardGroup>