Skip to main content
Collections in Arcbeam enable collaborative review workflows, allowing teams to work together on evaluating AI/LLM outputs, data quality, and application traces. Team members can be assigned as reviewers, submit approvals or rejections, and discuss findings through comments.

​
Overview

Collection reviews provide a structured workflow for quality assurance and peer review:
  • Assign reviewers to collections
  • Submit reviews with approval, rejection, or pending status
  • Add review comments to provide context and feedback
  • Track review progress with visual status indicators
  • Discuss findings through collection-level comments
This feature is ideal for:
  • Evaluating LLM output quality before production deployment
  • Peer reviewing data quality improvements
  • Collaborative debugging of RAG pipeline issues
  • Team-based validation of trace collections

​
Creating a Collection

Before you can review traces together, you need to create a collection. Collections are curated sets of traces organized around a specific theme or purpose.

​
From the Traces Page

1

Navigate to traces

Go to your project’s traces page to see all available traces.
2

Select traces

Use filters to find relevant traces, then select the ones you want to group together. You can filter by:
  • Date range
  • Search terms (e.g., “refund”, “error”)
  • Status codes
  • Model used
3

Create collection

Click Create Collection button with your selected traces.
4

Name and describe

Give your collection a descriptive name and explanation:
  • Good: “GPT-4 Response Quality - Week 12”
  • Good: “Refund Policy Errors - Dec 2024”
  • Avoid: “Collection 1” or “Test”
Add a description explaining the purpose:
"Evaluating production traces from the new checkout flow to ensure
AI responses are accurate and helpful. Focus areas: refund policy
clarity and return process guidance."
5

Save collection

Click Save to create the collection. It will now appear in your project’s collections list.

​
From Individual Traces

While viewing a specific trace:
  1. Click Add to Collection button
  2. Choose an existing collection or create a new one
  3. The trace is immediately added to that collection

​
Using Filters to Auto-Create Collections

Save time by creating collections directly from filter results:
1

Apply filters

Use the traces filter panel to find specific patterns. For example:
  • Date range: Last 7 days
  • Search: “refund”
  • Status: 200
2

Save as collection

Click Save as Collection to turn all matching traces into a collection.
3

Optional auto-sync

Choose whether new traces matching these filters should automatically be added to the collection.

​
Collection Types and Use Cases

Bug Investigation
  • Group all traces related to a specific bug
  • Example: “Checkout calculation errors - Issue #234”
Feature Testing
  • Collect traces showing a new feature in action
  • Example: “Multi-language support beta testing”
User Feedback Review
  • Traces where users gave negative feedback
  • Example: “Thumbs down responses - December”
Topic Analysis
  • All traces about a specific subject
  • Example: “Pricing and billing questions”
Model Comparison
  • Compare different models or prompts
  • Example: “GPT-4o vs GPT-4o-mini - Customer Support”

​
Adding Reviewers to a Collection

1

Navigate to the collection

Open the collection you want to add reviewers to from your project’s collections list.
2

Click the assignees section

In the collection header, you’ll see an assignees area showing current members. Click the ”+ Add” button or assignee avatars to manage members.
3

Select team members

Choose one or more team members from your organization to add as reviewers.
4

Members are added

Added members will appear with their status set to “Pending” until they submit their review.
Only collection creators can add or remove members from collections.

​
Submitting a Review

Once assigned to a collection, you can submit your review:
  1. Review the traces: Navigate through the collection’s traces to evaluate the quality, accuracy, and performance
  2. Add comments: Use the Comments tab to discuss specific findings or ask questions
  3. Submit your review: Click your avatar in the assignees section and select one of:
    • Approve: The collection meets quality standards
    • Request changes: Issues found that need addressing
    • Mark as pending: Still reviewing or awaiting additional information
  4. Add a review comment: Provide context for your decision to help the team understand your reasoning

​
Review Status Indicators

Collection members are displayed with visual indicators:
  • 🟢 Green checkmark: Review approved
  • 🔴 Red X: Changes requested or rejected

​
Review Workflow Example

Here’s a typical review workflow for evaluating LLM outputs:
1

Create collection

A data scientist creates a collection called “GPT-4 Response Quality - Week 12” containing 50 production traces.
2

Assign reviewers

Two team members are added as reviewers: a senior engineer and a domain expert.
3

Review traces

Each reviewer examines the traces, looking for hallucinations, off-topic responses, or formatting issues.
4

Add feedback

Reviewers add comments on specific traces they find problematic:
“Trace #127 shows hallucination - the model cited a non-existent research paper”
“Overall quality looks good, but 3 traces had formatting inconsistencies”
5

Submit reviews

The senior engineer approves with a comment: “Minor issues, but acceptable for production”The domain expert requests changes: “Need to address the hallucination in trace #127 before deploying”
6

Address feedback

The data scientist filters out the problematic trace, adjusts the prompt, and updates the collection.
7

Re-review

After fixes, the domain expert changes their review to approved.

​
Managing Collection Members

​
View All Members

To see all members assigned to a collection:
  1. Open the collection detail page
  2. The assignees section shows all current members with their review status
  3. Hover over any member to see their role and review comment (if provided)

​
Remove a Member

To remove a member from a collection:
  1. Click on the member’s avatar in the assignees section
  2. Select “Remove from collection”
  3. Confirm the removal

​
Using Comments for Discussion

The Comments tab in a collection provides a threaded discussion space for reviewers:

​
Adding a Comment

  1. Navigate to the Comments tab in the collection
  2. Type your comment in the text field (max 1000 characters)
  3. Press Cmd+Enter (Mac) or Ctrl+Enter (Windows) to submit
  4. Your comment appears with your name and avatar

​
Comment Best Practices

  • Be specific: Reference trace IDs when discussing specific issues
  • Provide context: Explain why something is problematic or noteworthy
  • Ask questions: Use comments to clarify requirements or get additional input
  • Link to traces: Mention trace IDs so team members can easily find what you’re discussing
Example good comment: 
Reviewing traces 45-67, I noticed the retrieval quality dropped significantly.
The similarity scores are < 0.5 compared to our usual 0.7+ range.

@sarah Should we adjust the embedding model parameters or is this expected
for this data source?

​
Collection Status Management

Collections have four status levels that help teams track progress:
  • Open: Collection is actively being reviewed
  • In Progress: Review is underway but not yet complete
  • Completed: All reviews submitted and any issues resolved
  • Closed: Final status, collection archived
Change the collection status from the collection detail page to reflect the current stage of review.

​
Tips for Effective Reviews

Batch similar traces: Group traces with similar patterns into collections for more efficient review. Instead of reviewing 500 random traces, create focused collections like “Error Cases - Week 12” or “High-Cost Queries - Dec 2024”.
Set review deadlines: Communicate expected review turnaround times to your team, especially for production-critical evaluations.
Review incrementally: For large collections, assign different sections to different reviewers to parallelize the work.

​
Next Steps