> ## Documentation Index > Fetch the complete documentation index at: https://docs.arcbeam.ai/llms.txt > Use this file to discover all available pages before exploring further. # Data Processing > How to prepare your information for AI **Data processing** is the work of transforming your documents, files, and information into a format that AI can understand and use effectively. Think of it like organizing a library so people can find exactly what they need. ## Why Data Processing Matters Imagine you have a thousand-page employee handbook as one giant PDF. When someone asks "What's the vacation policy?", the AI needs to find just the relevant section about vacations, not the entire handbook. Large documents get split into smaller, manageable pieces that are easier to search and retrieve Organized content allows the AI to find exactly the right information quickly without searching through everything Content is structured logically so related information can be found together Labels and categories help filter and locate the most relevant information for each question ## The Data Processing Pipeline This is where you collect all the information your AI needs to work with. **What you might collect**: PDF documents like manuals, guides, and reports Word documents, presentations, and spreadsheets Pages from your website, help center, or internal wiki **Things to consider**: Is this information current and accurate? Do you have permission to use this data? Is the quality good enough (not blurry scans, readable text)? Just like you'd remove coffee stains from a book before scanning it, you need to clean your data. **What gets cleaned**: Page numbers, headers, and footers that aren't useful for answering questions Extra spaces, line breaks, and special characters that don't add meaning Repeated information that could confuse the AI or waste storage Copyright notices, legal boilerplate, and other non-essential content Cleaner data means better, more relevant answers from your AI This is one of the most important steps. Large documents get split into smaller, meaningful sections. **Think of it like this**: Instead of having one book, you have individual chapters Instead of a whole webpage, you have distinct paragraphs or sections Each piece is small enough to be useful on its own **How big should pieces be?** | Chunk Size | Good For | Benefit | Drawback | | -------------------------------------- | ------------------------------------------ | ------------------------- | ------------------------------ | | **Smaller pieces** (a few paragraphs) | Specific facts, product specs, FAQ answers | Very precise answers | Might miss surrounding context | | **Larger pieces** (several paragraphs) | Explanations, how-to guides, policies | More complete information | Takes longer to search through | Most teams start with sections that are about 2-3 paragraphs long, then adjust based on results This step converts your text into a special format that computers can search through very quickly. **The simple explanation**: Each piece of text gets converted into a series of numbers (a mathematical representation) Similar content gets similar numbers, making it easy to find related information When someone asks a question, the AI converts that question to numbers too The system finds the content with the most similar numbers to the question It's like creating a fingerprint for each piece of content. Questions about "vacation time" will match content about "paid time off" even though the words are different. **What you need to know**: This process has a small cost (usually pennies per thousand pieces) It happens automatically without manual intervention The quality of this step affects how well your AI finds relevant information This is like adding sticky notes to your content with helpful information. **What you might add**: **Source**: "Product Manual v2.0" - Tracks which document the content came from **Date**: "Last updated: January 2024" - Helps identify freshness of information **Category**: "Installation Guide" - Groups similar types of content together **Product**: "Widget Pro" - Links content to specific products or services **Department**: "Engineering" - Identifies which team owns this content **Why this helps**: You can filter results to only show recent content or specific categories You can organize by product, topic, or department for easier navigation You know when content might be outdated and needs updating You can trace answers back to their original source for verification Finally, all your processed content gets stored in a vector database where it can be quickly searched. **What happens**: Each piece is stored with its converted format and all metadata labels The database organizes everything for lightning-fast searching You can easily add new content or update existing content without disruption ## Common Processing Decisions ### How to Split Your Documents **Best Practice**: Keep each question and answer together as one piece **Don't split them apart** - Example: "Q: How do I reset my password?" + "A: Click the forgot password link..." should stay together **Best Practice**: Split by sections or headings **Keep related paragraphs together** and make sure each piece can stand alone with enough context **Best Practice**: Split by policy topic or clause **Include the policy title** with each piece and keep numbered sections together for coherence **Best Practice**: Keep all specs for one product together **Include product name** in each piece and don't split feature lists across multiple pieces ### Handling Different File Types **Text PDFs**: Usually process easily with standard extraction **Scanned PDFs**: May need extra processing (OCR) to extract text from images **Forms and tables**: Might need special handling to preserve structure **Generally easy to process** with good text extraction **Headers and styles** help with automatically splitting into logical sections **Images and charts** may need separate handling or description **Remove navigation**, ads, and boilerplate content that doesn't add value **Keep the main content** that answers questions or provides information **Preserve links** that add context or point to related information **Convert to a readable format** first (like CSV or structured text) **Each row might become one piece** of information to search **Include column headers** for context so the data makes sense ## Best Practices ### Start Simple Select your most important 10-20 documents that will answer the majority of common questions Start with default settings for splitting - your tool will have recommendations based on best practices Process your documents and test with real questions people actually ask See what works well and what doesn't - gather feedback from actual usage Make targeted adjustments based on what you learned, then repeat **Don't try to be perfect**: Get something working first, then improve it. Iteration beats perfection. ### Test and Iterate **After processing your data**: Ask questions you already know the answers to - this helps verify accuracy See if the AI finds the right information from your content Check if answers have enough surrounding context to be useful Look for cases where it finds the wrong or irrelevant information **Common fixes**: **Problem**: Information exists but isn't being retrieved **Solution**: Add more content or check your metadata labels for accuracy **Problem**: Responses are too brief or missing important details **Solution**: Make your content pieces bigger to include more surrounding information **Problem**: Responses aren't specific enough to be actionable **Solution**: Make your pieces smaller for more precise, focused answers **Problem**: AI returns content from wrong section or document **Solution**: Check your categories and labels - improve organization and tagging ## Monitoring Processing Quality ### Signs Your Processing is Working Well Users consistently get accurate, relevant answers to their questions The AI finds information quickly without long search times Answers include the right amount of detail - not too brief, not too verbose Users rarely say "that's wrong" or "I can't find what I need" ### Signs You Need to Adjust AI often says "I don't know" even though you have the information in your content Answers are missing important context or background information to be useful AI returns information from the wrong section or document instead of relevant content Responses are slow or processing costs are higher than expected ## Troubleshooting Common Issues **Likely cause**: Your pieces are too small and don't include enough context **Fix**: Make your pieces bigger or add overlap between pieces so related information appears together **Likely cause**: Your categories and tags aren't specific enough **Fix**: Add more specific tags and improve your organization with better metadata to help filter results **Likely cause**: You're processing too much at once or pieces are too numerous **Fix**: Process in batches, or make pieces slightly larger to reduce total count and improve efficiency **Likely cause**: You haven't updated your processed data recently **Fix**: Set up regular update schedule and version your content to ensure freshness ## Getting Started ### Your First Processing Project * Identify your 10 most important documents * Make sure they're current and accurate * Gather them in one location * Choose a vector database * Run your first processing job * Review the results * Ask 20-30 test questions * Evaluate the answers * Note what works and what doesn't * Adjust your settings based on results * Re-process if needed * Add more documents ### Questions to Ask Your Technical Team If you're working with engineers or technical staff: "How big are our chunks? Should we try different sizes based on our content type?" "How often will we refresh the data? What's our update schedule?" "Can we set up automated updates to keep content fresh without manual work?" "What categories or tags should we use to organize our content effectively?" "How will we know if processing is working well? What should we measure?" ## Next Steps Understand where your data comes from Choose the right AI model for your needs Test if your processing is working well Track where your information comes from