Compare Different Versions

​

What to Compare

• Different models (GPT-4 vs GPT-4o-mini, Claude vs GPT)
• Prompt variations (different instructions or examples)
• Retrieval strategies (different k values, embedding models)
• System configurations (temperature, max tokens, etc.)
• Feature changes (new vs old implementation)

​

Setting Up Comparisons

​

Tag Traces with Versions

from arcbeam_connector.langchain.connector import ArcbeamLangConnector

# Version A
connector = ArcbeamLangConnector(
    base_url="http://platform.arcbeam.ai",
    api_key="your-api-key-here",
    project_id="your-project-id-here",
    environment="v2.1.0-gpt4o-mini",
)
connector.init()

​

Run A/B Tests

import random
from arcbeam_connector.langchain.connector import ArcbeamLangConnector
from langchain_openai import ChatOpenAI

if random.random() < 0.5:
    # Version A
    connector = ArcbeamLangConnector(
        base_url="http://platform.arcbeam.ai",
        api_key="your-api-key-here",
        project_id="your-project-id-here",
        environment="version-a-gpt4",
    )
    connector.init()
    model = ChatOpenAI(model="gpt-4")
else:
    # Version B
    connector = ArcbeamLangConnector(
        base_url="http://platform.arcbeam.ai",
        api_key="your-api-key-here",
        project_id="your-project-id-here",
        environment="version-b-gpt4o-mini",
    )
    connector.init()
    model = ChatOpenAI(model="gpt-4o-mini")

​

Comparing Results

​

Create Collections

1. Filter traces by version: "a"
2. Create collection “Version A Results”
3. Repeat for Version B
4. Review side-by-side

​

Compare Metrics

​

Statistical Significance

• Run at least 100 traces per version
• Check if differences are meaningful
• Consider variance and outliers

​

Common Comparisons

​

Model Comparison

1. Run same queries through both models
2. Tag with model: "gpt-4" and model: "gpt-4o-mini"
3. Compare:
   • Quality (user feedback, accuracy)
   • Cost (GPT-4 is more expensive)
   • Speed (GPT-4o-mini is faster)
4. Decide based on requirements

• GPT-4o-mini: 90% of GPT-4 quality at 10% of the cost
• Decision: Use GPT-4o-mini for most queries, GPT-4 for complex ones

​

Prompt Comparison

Answer the user's question based on the provided context.

You are a helpful assistant. Use only the provided context to answer.
If the context doesn't contain the answer, say so clearly.
Be concise and accurate.

1. Deploy both prompts (A/B split)
2. Tag with prompt_version
3. Compare user satisfaction and accuracy
4. Choose winner

​

Retrieval Strategy Comparison

1. Version A: k=3
2. Version B: k=5
3. Compare:
   • Answer quality
   • Token usage (more docs = more tokens)
   • Latency
4. Find optimal balance

• k=5 improved quality by 5%
• But increased cost by 25%
• Decision: Use k=3 for simple queries, k=5 for complex ones

​

Embedding Model Comparison

1. Re-embed dataset with different models:
   • text-embedding-ada-002
   • text-embedding-3-large
2. Run same queries against both
3. Compare relevance scores and answer quality
4. Choose best performer

​

Analyzing Differences

​

Side-by-Side Comparison

1. Open trace from Version A
2. Find corresponding trace from Version B (same input)
3. Compare:
   • Outputs (quality, length, tone)
   • Retrieved documents
   • Costs
   • Timing

​

Aggregate Statistics

Version A (GPT-4):
- Total traces: 500
- Success rate: 98%
- Avg cost: $0.045
- Avg latency: 2.3s
- Avg user rating: 4.5/5

Version B (GPT-4o-mini):
- Total traces: 500
- Success rate: 95%
- Avg cost: $0.008
- Avg latency: 1.1s
- Avg user rating: 4.2/5

​

Qualitative Review

1. Read actual outputs from both versions
2. Check for subtle quality differences
3. Look for edge cases where one fails
4. Get feedback from stakeholders

​

Making Decisions

​

Define Success Criteria

1. Quality (most important)
2. Cost (important)
3. Speed (nice to have)

1. Cost (most important)
2. Quality (important)
3. Speed (less important)

​

Calculate ROI

• GPT-4o-mini saves $10,000/month
• But quality drops slightly (4.2 vs 4.5 rating)
• Question: Is the quality drop worth $10k savings?

​

Gradual Rollout

1. Start with 10% traffic to new version
2. Monitor for issues
3. Gradually increase (25%, 50%, 75%)
4. Full rollout only after validation

​

Best Practices

​

Use Consistent Test Queries

test_queries = [
    "What is the refund policy?",
    "How do I cancel my subscription?",
    "What payment methods do you accept?"
]

for query in test_queries:
    # Run through both versions and compare results in the Arcbeam dashboard
    # Filter by environment tag to separate the versions
    result = qa_chain.invoke(query)

​

Document Assumptions

Version A: GPT-4, temp=0, k=3, prompt v2
Version B: GPT-4o-mini, temp=0, k=5, prompt v3

Hypothesis: GPT-4o-mini with more context (k=5) will match
GPT-4 quality at lower cost.

​

Avoid Contamination

• Use same time period (avoid external factors)
• Same data sources
• Same user base (random split)
• Control all variables except what you’re testing

​

Set Time Limits

• Small changes: 1-3 days
• Major changes: 1-2 weeks
• Collect enough data, then decide

​

Next Steps