> ## Documentation Index
> Fetch the complete documentation index at: https://docs.arcbeam.ai/llms.txt
> Use this file to discover all available pages before exploring further.
# PGvector Integration
> Connect PostgreSQL with PGvector to Arcbeam for data lineage tracking
Connect your PostgreSQL database with PGvector extension to Arcbeam to enable full data lineage tracking. See which documents influence each AI response and track data usage over time.
## What Is PGvector?
[PGvector](https://github.com/pgvector/pgvector) is a PostgreSQL extension for vector similarity search. It's the most popular open-source vector database solution, especially for self-hosted applications.
PGvector allows you to:
* Store vector embeddings alongside your data
* Perform similarity searches using various distance metrics
* Scale to millions of vectors
* Keep everything in PostgreSQL (no additional infrastructure)
## Why Connect PGvector to Arcbeam?
Link AI traces to the exact documents that influenced them. Trace wrong answers back to outdated documents.
Track which documents are retrieved most often, which are never used, and retrieval patterns over time.
Demonstrate data provenance with full audit trail from output to source.
## Prerequisites
Before connecting PGvector to Arcbeam, you need:
### 1. PostgreSQL with PGvector
Install the PGvector extension:
```sql theme={null}
CREATE EXTENSION vector;
```
### 2. Table with Vector Data
A table containing your documents and embeddings:
```sql theme={null}
CREATE TABLE documents (
id UUID PRIMARY KEY,
content TEXT,
embedding VECTOR(1536), -- Dimension depends on your model
source TEXT,
title TEXT,
updated_at TIMESTAMP DEFAULT NOW()
);
```
### 3. Database Credentials
Read access to the database:
```
postgresql://username:password@host:port/database
```
### 4. Arcbeam Project
An existing project where you'll connect the dataset.
## Connecting PGvector
Go to **Datasets** page in Arcbeam and click **New Dataset**
Choose **PGvector** as the dataset type
* **Connection String**: `postgresql://user:pass@host:port/db`
* **Table Name**: Name of your documents table (e.g., `documents`)
* **ID Column**: Unique identifier (e.g., `id`)
* **Content Column**: Document text (e.g., `content`)
* **Embedding Column**: Vector field (e.g., `embedding`)
* **Metadata Columns**: Optional fields like `source`, `title`, `updated_at`
Click **Test Connection** to verify, then **Create Dataset**
Your PGvector database is now connected to Arcbeam!
```bash theme={null}
curl -X POST https://api.arcbeam.ai/v1/vectordb-integrations \
-H "Authorization: Bearer YOUR_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"name": "Product Documentation",
"description": "PGvector integration for product docs",
"type": "pgvector",
"hostUrl": "postgresql://user:pass@host:5432/db",
"indexName": "documents",
"schemaMapping": {
"idField": "id",
"documentField": "content",
"sourceField": "source",
"embeddingField": "embedding",
"lastUpdatedField": "updated_at"
},
"isActive": true
}'
```
Use the API method for programmatic setup or infrastructure-as-code deployments.
## Syncing Data
Arcbeam periodically syncs metadata from your PGvector database.
### Manual Sync
Trigger a sync anytime:
1. Go to **Datasets** page
2. Find your PGvector dataset
3. Click **Sync Now**
4. Wait for completion (usually \< 1 minute for small datasets)
### What Gets Synced
| Data Type | Synced? | Description |
| ---------------- | ----------- | --------------------------------------- |
| Document IDs | ✓ Yes | To match retrieved docs in traces |
| Content | ✓ Yes | For display in trace details |
| Metadata columns | ✓ Yes | Only fields you explicitly map |
| **Embeddings** | ✗ **Never** | Not needed, saves storage and bandwidth |
| Other tables | ✗ **Never** | Only the table you specify |
| Unmapped fields | ✗ **Never** | Sensitive data unless explicitly mapped |
### Sync Performance
For large datasets:
* **\< 10k docs**: Sync in seconds
* **10k-100k docs**: Sync in minutes
* **100k+ docs**: Sync may take longer
## Security & Privacy
### Connection Security
* Connections use SSL/TLS encryption
* Credentials stored encrypted at rest
* Access limited to read-only queries
* No write access to your database
### Data Access
Arcbeam queries:
* Only the table you specify
* Only columns you map
* Read-only SELECT queries
Arcbeam does **not**:
* Modify your data
* Access other tables
* Store full embeddings
* Keep entire documents (only excerpts for display)
### Credentials Management
Best practices:
* Use read-only database user
* Limit access to specific table
* Rotate credentials periodically
* Use connection pooling limits
Example read-only user:
```sql theme={null}
CREATE USER arcbeam_readonly WITH PASSWORD 'secure_password';
GRANT CONNECT ON DATABASE mydb TO arcbeam_readonly;
GRANT USAGE ON SCHEMA public TO arcbeam_readonly;
GRANT SELECT ON documents TO arcbeam_readonly;
```
## What You Can Do
Once your PGvector database is connected, you can:
Trace incorrect AI responses back to the source documents and fix them
Find and remove unused documents to improve retrieval speed and quality
Identify high-traffic documents that need updating
Generate audit trails showing complete data provenance
[See detailed use cases and workflows →](/v0/data-insights/see-what-data-is-used)
## Best Practices
### Keep Source Attribution
Always include source field:
```sql theme={null}
source TEXT -- "refund_policy_v2024.pdf"
```
### Track Update Timestamps
Maintain `updated_at` for tracking:
```sql theme={null}
updated_at TIMESTAMP DEFAULT NOW()
```
Update on changes:
```sql theme={null}
UPDATE documents
SET content = new_content, updated_at = NOW()
WHERE id = document_id;
```
### Use Meaningful Source Names
Consistent, descriptive names:
```
✅ Good: "user_manual_2024.pdf", "api_docs_v3.html"
❌ Bad: "file1.pdf", "doc_final_FINAL_v2.pdf"
```
### Index for Performance
Create indexes for faster searches:
```sql theme={null}
-- Vector similarity index
CREATE INDEX ON documents USING ivfflat (embedding vector_cosine_ops);
-- Metadata indexes
CREATE INDEX ON documents(source);
CREATE INDEX ON documents(updated_at);
```
## Troubleshooting
### Connection Failed
**Check connection string format**:
```
postgresql://username:password@host:port/database
```
**Verify network access**:
* Can Arcbeam reach your database?
* Is your database behind a firewall?
* Consider using a VPN or allowlist Arcbeam IPs
**Test connection manually**:
```bash theme={null}
psql "postgresql://user:pass@host:port/db"
```
### Documents Not Linking
**Check ID column mapping**:
* Ensure ID column name is correct
* Verify IDs in traces match IDs in database
**Trigger manual sync**:
* Go to Datasets → Sync Now
* Wait for completion
* Check if documents now appear
**Verify data exists**:
```sql theme={null}
SELECT COUNT(*) FROM documents;
```
### Slow Syncs
**Optimize table**:
```sql theme={null}
VACUUM ANALYZE documents;
```
**Add indexes**:
```sql theme={null}
CREATE INDEX ON documents(id);
```
**Limit metadata columns**:
* Only sync fields you need
* Reduces sync time
## Next Steps
Step-by-step guide to connecting PGvector
Understand how lineage tracking works
Explore document usage analytics
Debug problems using lineage