> ## Documentation Index
> Fetch the complete documentation index at: https://docs.arcbeam.ai/llms.txt
> Use this file to discover all available pages before exploring further.
# Cost Optimization
> Reducing AI costs without sacrificing quality
**Cost optimization** is about making your AI system cost-effective while maintaining good quality. AI can get expensive quickly, but there are many ways to reduce costs without users noticing any difference.
Poor optimization can waste 50-80% of your AI budget
## Why AI Costs Matter
### AI Is Different from Traditional Software
Traditional software is like owning a car (fixed costs), while AI is like taking taxis everywhere (per-trip costs that add up).
| Traditional Software | AI Systems |
| --------------------------------- | -------------------------------------------- |
| **Fixed cost model** | **Variable cost model** |
| You pay for servers once | You pay per use (every question costs money) |
| Costs are predictable | Costs grow with usage |
| Scaling doesn't change costs much | More users = significantly higher costs |
| Budget is stable and forecastable | Poor optimization can waste 50-80% of budget |
## Where AI Costs Come From
### The Four Main Cost Categories
**60-70% of total costs**
* Every time you ask the AI a question, you pay
* Charged based on how much text it processes
* More expensive models cost more
* Longer conversations cost more
**15-25% of total costs**
* Converting documents to searchable format
* Happens when you add new content
* One-time cost per document
* Can add up with large document sets
**10-15% of total costs**
* Storing your documents
* Running search databases
* Server costs
* Network/bandwidth
**5-10% of total costs**
* Monitoring tools
* Development time
* Quality review
* Testing
### Example Monthly Breakdown
**A typical AI chatbot processing 10,000 questions/day**:
| Cost Category | Monthly Cost | Percentage |
| ----------------------- | ------------ | ---------- |
| **Total Monthly Cost** | **\$3,000** | **100%** |
| AI model calls | \$2,000 | 67% |
| Document processing | \$500 | 17% |
| Database/storage | \$400 | 13% |
| Other operational costs | \$100 | 3% |
## Strategies to Reduce Costs
### 1. Use Cheaper Models When Possible
Not every question needs the most expensive AI model.
#### Model Pricing Comparison
| Model Tier | Examples | Cost per 1,000 Questions | Best For |
| ------------ | -------------------------- | ------------------------ | ------------------------------------ |
| **Premium** | GPT-4, Claude Opus | \$5-15 | Complex analysis, critical decisions |
| **Mid-tier** | GPT-4o mini, Claude Sonnet | \$0.50-2 | Most use cases, balanced quality |
| **Basic** | GPT-3.5, Claude Haiku | \$0.20-0.50 | Simple FAQs, basic facts |
**Key insight**: Most questions (70-80%) are simple and work fine with cheaper models - that's a 20x cost reduction opportunity!
| Simple Question | Complex Question |
| ----------------------------------- | ---------------------------------------------------------------------------------------------------------------- |
| **"What are your business hours?"** | **"Explain the differences between your three subscription plans and recommend one based on my usage patterns"** |
| Use basic model | Use premium model |
| Cost: \$0.0001 | Cost: \$0.002 |
| Perfect for FAQs and simple facts | Needed for analysis and recommendations |
**Potential savings**: 50-80% of model costs by routing questions to appropriate models
### 2. Keep Conversations Shorter
The AI reads the entire conversation history each time it responds. Long conversations get expensive.
| Turn | Without Optimization | With Optimization |
| ----------- | ----------------------------- | ---------------------- |
| **Turn 1** | 200 words → \$0.001 | 200 words → \$0.001 |
| **Turn 5** | 1,000 words → \$0.005 | 400 words → \$0.002 |
| **Turn 10** | 2,000 words → \$0.010 | 600 words → \$0.003 |
| **Turn 20** | 4,000 words → \$0.020 | 800 words → \$0.004 |
| **Result** | Cost doubles every few turns! | Costs stay manageable! |
**Optimization strategies**:
* Summarize old messages (keep last 5-10 in detail)
* Remove unnecessary context
* Start fresh after certain time/messages
* Don't repeat information
**Potential savings**: 60-70% on long conversations
### 3. Limit Response Length
Longer AI responses cost more. Most users don't need 500-word answers.
**Question**: "What's your refund policy?"
| Aspect | Unoptimized Response | Optimized Response |
| -------------- | --------------------------------------------------- | ------------------------------------------------- |
| **Word count** | 500 words | 100 words |
| **Content** | Detailed explanation, multiple examples, edge cases | Concise answer, key points only, clear and direct |
| **Cost** | \$0.005 | \$0.001 |
| **Savings** | - | 80% per question |
**Win-win optimization**: Users prefer shorter, clearer answers anyway - lower costs AND better user experience!
**Implementation tips**:
* Set reasonable length limits (100-200 words for most answers)
* Ask AI to be concise in your system prompts
* Test to ensure quality isn't sacrificed
**Potential savings**: 50-70% on output costs
### 4. Cache Common Questions
If people ask the same questions repeatedly, save and reuse answers.
**Scenario**: 10 people ask "What are your business hours?"
| Metric | Without Caching | With Caching |
| ------------------- | ------------------------- | ----------------------------- |
| **First request** | \$0.001 | \$0.001 (generate & save) |
| **Next 9 requests** | \$0.001 each | \$0.000 (return saved) |
| **Total cost** | \$0.01 | \$0.001 |
| **Computation** | Same computation 10 times | Computed once, reused 9 times |
| **Savings** | - | 90% |
**What to cache**:
* FAQs (e.g., "How do I reset my password?")
* Common product questions
* Policies and procedures
* Anything asked multiple times
**Cache freshness guidelines**:
* Keep cached answers for 1-24 hours depending on content type
* Update when information changes
* Review cache hit rate to measure effectiveness
**Typical impact**: 20-40% overall cost reduction
### 5. Search Fewer Documents
When AI searches your knowledge base, each document adds cost.
| Step | Expensive Approach | Optimized Approach |
| -------------- | ---------------------------- | ----------------------- |
| **Search** | Search for 20 documents | Search for 20 documents |
| **Selection** | Include all 20 in AI context | Pick best 5 to include |
| **Word count** | 8,000 words | 2,000 words |
| **Cost** | \$0.020 | \$0.005 |
| **Savings** | - | 75% |
**Quality over quantity**: 5 well-chosen documents often provide better answers than 20 mediocre ones
**Document selection guidelines**:
* Start with 3-5 documents
* Only increase if quality suffers
* Use relevance scoring to pick the best matches
* Test to find the optimal number for your use case
**Potential savings**: 60-75% on retrieval costs
### 6. Use Cheaper Document Processing
Converting text to searchable format costs money. Use efficient methods.
#### Embedding Model Cost Comparison
| Model Type | Cost per Million Words | Best For |
| ---------------------- | ---------------------- | ---------------------------------------------- |
| Premium embedding | \$0.13 | Specialized domains requiring highest accuracy |
| **Standard embedding** | **\$0.02** | **95% of use cases (recommended)** |
| Basic embedding | \$0.10 | Simple keyword matching |
**Recommendation**: Standard embedding models work for 95% of use cases at a fraction of the cost
**Additional optimization strategies**:
* Remove duplicate content before processing
* Don't re-process unchanged documents
* Batch process instead of one-at-a-time
* Use incremental updates for document changes
**Typical savings**: 30-40% on document processing costs
### 7. Optimize Database Costs
Your vector database doesn't need to be oversized.
| Metric | Over-Provisioned Database | Right-Sized Database |
| ---------------- | --------------------------------- | ----------------------------------- |
| **Monthly cost** | \$500 | \$100 |
| **Capacity** | Handles 1M queries/day | Handles 50k queries/day |
| **Actual usage** | 10k queries/day | 10k queries/day |
| **Utilization** | 1% (99% unused capacity) | 20% (plenty of headroom for growth) |
| **Efficiency** | Wasting money on unused resources | Optimized for actual needs |
| **Savings** | - | \$400/month |
**Questions to guide right-sizing**:
* How many searches per day do you actually need?
* How much data are you storing?
* What's your growth projection for the next 6-12 months?
* Are you using a managed service when self-hosted would work?
**Potential savings**: 30-60% on infrastructure costs
### 8. Compress and Archive Old Data
Not all data needs to be instantly accessible.
| Aspect | Active Data (Last 3 Months) | Archived Data (Older) |
| ------------------ | --------------------------------------- | ------------------------------ |
| **Storage type** | Fast database | Cheap cold storage |
| **Monthly cost** | \$200 | \$20 |
| **Access pattern** | Used frequently | Accessed rarely |
| **Performance** | Needs quick access, optimized for speed | Slower retrieval is acceptable |
| **Savings** | - | \$180/month |
**Archiving best practices**:
* Archive data after 3-6 months of inactivity
* Compress before archiving to save additional storage costs
* Keep a lightweight search index for archived data
* Set up retrieval process for rare access needs (slower but acceptable)
## Monitoring and Tracking Costs
### Setting Up Alerts
| Alert Type | Trigger | Action Required |
| --------------------- | ------------------------------- | ---------------------------- |
| **Budget Alerts** | | |
| Daily budget exceeded | Spending > daily limit | Immediate action needed |
| High daily spending | Approaching 80% of daily budget | Warning - review today |
| Weekly overspend | Week running 20% over expected | Review needed |
| Monthly trend | Month trending over budget | Time to optimize |
| **Pattern Alerts** | | |
| Expensive query | Single query cost > \$1 | Investigate this query |
| Cost spike | Average cost increased 50% | Something changed - review |
| Feature spike | Specific feature spiking | Potential issue - check logs |
| Cache degradation | Cache hit rate dropped | Check cache config |
### Understanding Your Costs
#### Example: Cost Breakdown by Query Type
| Query Type | Monthly Cost | Percentage | Action |
| ----------------- | ------------ | ---------- | ---------------------------------------- |
| Product questions | \$800 | 40% | **Optimize first** - biggest cost driver |
| Support questions | \$600 | 30% | Second priority for optimization |
| General chat | \$400 | 20% | Consider limiting conversation length |
| Other | \$200 | 10% | Monitor for patterns |
**Key insight**: Focus optimization efforts on the highest-cost categories first for maximum impact
#### Example: Cost Breakdown by User Segment
| User Type | Monthly Cost | Percentage | Consideration |
| ---------- | ------------ | ---------- | ------------------------------------------- |
| Free users | \$1,200 | 60% | Consider usage limits or conversion prompts |
| Paid users | \$800 | 40% | Ensure quality experience is maintained |
Analyzing costs by user segment helps you make informed decisions about feature access and pricing tiers
## Cost Optimization Checklist
### Quick Wins (Week 1)
**Easy changes - Start here for immediate impact**
Use mid-tier or basic models for FAQ-style questions instead of premium models. Most questions (70-80%) don't need the most expensive AI.
Implement caching for common questions to avoid regenerating the same answers repeatedly.
Limit output length to reduce costs. Users prefer concise answers anyway.
Decrease the number of documents included in context. Quality over quantity.
Switch from premium to standard embedding models - they work for 95% of use cases.
**Expected savings**: 40-60% reduction in costs
### Medium Effort (Weeks 2-4)
**More involved optimizations for additional savings**
Summarize long conversations to reduce context size and token usage
Build smart caching system for common query patterns
Route queries through cache → simple model → complex model hierarchy
Optimize database resources to match actual usage patterns
Clean up duplicate content before processing
Process new documents in batches instead of one-at-a-time
**Expected additional savings**: 20-30% reduction in costs
### Advanced (Months 2-3)
**Sophisticated optimizations for mature systems**
Automatically classify query complexity to route to appropriate model tiers
Try cheap model first, upgrade to premium only if needed
Move old, rarely-accessed data to cold storage
Fine-tune database indexes for better performance
Train specialized models for specific high-volume tasks
**Expected additional savings**: 10-20% reduction in costs
## Calculating Expected Costs
### Simple Cost Estimation
**Questions to answer for your estimation**:
1. How many queries per day? (example: 10,000)
2. Average question length? (example: 50 words)
3. Average answer length? (example: 150 words)
4. Documents needed per query? (example: 5)
5. Which model? (example: GPT-4o mini)
### Example Calculation Walkthrough
**Scenario**: 10,000 queries/day using GPT-4o mini
| Component | Calculation | Result |
| ----------------------- | ------------------------ | -------------- |
| **Per Query Breakdown** | | |
| Question | 50 words = \~70 tokens | 70 tokens |
| Context (5 documents) | 500 words = \~650 tokens | 650 tokens |
| Answer | 150 words = \~200 tokens | 200 tokens |
| **Total per query** | | **920 tokens** |
| | | |
| **Cost Breakdown** | | |
| Input cost | 720 tokens at \$0.15/1M | \$0.0001 |
| Output cost | 200 tokens at \$0.60/1M | \$0.0001 |
| **Cost per query** | | **\$0.0002** |
| | | |
| **Scaling** | | |
| Daily cost | 10,000 × \$0.0002 | \$2 |
| Monthly AI calls | \$2 × 30 days | \$60 |
| Infrastructure | Database + hosting | \$150 |
| **Total monthly** | | **\$210** |
### Comparing Scenarios
| Scenario | Optimizations | Monthly Cost | Savings | % Saved |
| ---------------------------- | ---------------------------------------------------------------------------------------------------------- | ------------ | ------- | ------- |
| **A: No optimization** | Premium model for everything
10 documents per query
No caching
No length limits | \$3,000 | - | - |
| **B: Basic optimization** | Mid-tier model for most queries
5 documents per query
Cache common questions
200-word limit | \$800 | \$2,200 | 73% |
| **C: Advanced optimization** | Smart model routing
3-5 documents (optimized)
Aggressive caching
Conversation summarization | \$400 | \$2,600 | 87% |
## Common Mistakes
| Mistake | Why It's Wrong | Better Approach |
| ----------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------- |
| **Using Premium Models for Everything**
"We'll just use GPT-4 for all queries to ensure quality" | 80% of queries are simple and don't need premium models - you're paying 20x more for minimal quality gain | Use mid-tier for most queries, premium only when needed. Test to see if users notice any difference |
| **Not Monitoring Costs**
Set up AI, never check costs until the bill arrives | Costs can spiral quickly and small issues become expensive problems | Daily cost monitoring, weekly reviews, monthly analysis - catch issues early |
| **Optimizing Without Measurement**
"This should reduce costs" without testing actual impact | You don't know if optimization worked or if you hurt quality | Measure before and after, track both costs AND quality metrics |
| **Sacrificing Quality for Cost**
Make AI so cheap it becomes useless | Users leave, defeating the entire purpose | Find the right balance - cut costs where users don't notice, preserve quality where they do |
These mistakes can waste 50-80% of your AI budget or harm user experience. Always balance cost optimization with quality maintenance.
## Getting Started
**Measure your baseline**:
* What's your current monthly bill?
* Cost per query?
* Most expensive query types?
* Where is money going?
**Easy optimizations**:
* Switch to mid-tier model
* Add response length limits
* Cache common questions
* Reduce retrieval documents
**Measure impact**:
* Did costs decrease?
* By how much?
* Any quality issues?
**Track results**:
* Cost savings achieved
* User satisfaction maintained?
* Any new issues?
* Where to optimize next?
**Set up ongoing optimization**:
* Regular cost reviews
* Budget alerts
* Quality monitoring
* Continuous improvement
## Next Steps
Choose cost-effective models for your use case
Manage context to reduce token usage
Monitor costs and identify optimization opportunities
Optimize data processing for efficiency