TL;DR: I built a multi-agent system with 20+ Claude Code instances working in parallel to generate high-quality training data for terminal-based coding tasks, producing 331+ validated datapoints for my scalable RL training project.

Why build an agentic data pipeline?

Because generating quality training data for coding agents requires creativity, validation, and scale—tasks perfectly suited for AI collaboration.

Watch 20+ agents working in parallel to generate training data at scale:

• See It In Action
• High-Level Architecture
• Pipeline Results
• Agent Roles & Workflow
  • Idea Generation Agents
  • Datapoint Builder Agents
  • Quality Review Agents
• Task Manager System
• Datapoint Structure
• Validation Pipeline
• Infrastructure & Tools
• Getting Started
• Key Design Decisions

The pipeline transforms Terminal Bench evaluation tasks into diverse training datapoints through three specialized agent stages:

1. Seed tasks from Terminal Bench → Idea Agents → Creative variations
2. Draft ideas → Builder Agents → Complete executable datapoints
3. Built datapoints → Review Agents → Production-ready training data

All agents work independently in parallel, coordinated by a central Task Manager that prevents duplication and handles failures.

• 331 validated datapoints generated
• 20+ agents working concurrently
• 3 specialized agent types with distinct roles
• 100% Docker-validated environments

• Python: Used in 196 datapoints (59%)
• CLI Tools: 47 datapoints
• APIs: 30 datapoints
• C: 22 datapoints

Each agent type operates independently with specialized tools and clear responsibilities:

• Input: Seed tasks from Terminal Bench
• Process: Analyze core skills → Generate n×multiplier variations → Select best ideas
• Output: Draft specifications in shared workspace

Key Innovation: Refinement criteria provided only AFTER brainstorming to maximize creativity

• Input: Draft specifications from idea agents
• Process: Build complete scenarios → Validate via software script → Iterate until passing
• Output: Executable datapoints with all required components

Validation Requirements:

• ✅ Dockerfile builds successfully
• ✅ Tests fail before agent intervention
• ✅ All dependencies present
• ✅ Test weights sum to 1.0

• Input: Validated datapoints from builders
• Process: Check quality standards → Edit if needed → Re-validate → Categorize
• Output: Approved datapoints with metadata or rejection with reasons

The Task Manager enables parallel agent coordination without complex handoffs:

# Agent claims work atomically
task = tm.get_next_task("idea-agent-07", task_types=["generate_idea"])

# Process independently
draft_ideas = generate_creative_variations(task["seed_data"])

# Complete with results
tm.complete_task(task["id"], "idea-agent-07", {"drafts": draft_ideas})

Features:

• Atomic task claiming (no collisions)
• Automatic timeout recovery
• Parent-child task tracking
• Real-time status monitoring

Each training datapoint contains:

draft_001_a/
├── prompt.md       # 1-3 sentence task (e.g., "Auth times out with 100+ users. Fix it.")
├── dockerfile      # Ubuntu 24.04 (or similar) environment setup
├── tests.py        # Pytest verification functions
├── weights.json    # Test importance distribution
└── files/          # Additional resources
    ├── app.py      # Broken code to fix
    └── config.json # Configuration files

Shared validation tools ensure quality across all agents:

1. Docker Build: Environment must build successfully
2. Test Discovery: Pytest must find all test functions
3. Fail-First: Tests must fail in initial state
4. Dependency Check: All required packages present
5. Weight Validation: Test weights sum to exactly 1.0

agents/
├── idea_agent_workspace/      # Idea generation tools & instructions
├── dp_builder_workspace/      # Building tools & staging areas
└── review_agent_workspace/    # Review tools & quality checks

shared_workspace/              # Common filesystem for all agents
shared_tools/                  # Validation, patching, utilities
task_manager/                  # Coordination & state management

• Idea Agents: get_task_parameters.py, get_idea_refinement_details.py
• Builder Agents: create_dp.py, add_dp_to_review.py
• Review Agents: approve_datapoint.py, cancel_datapoint.py, show_categories_tags.py

• validate_datapoint.py - Complete validation suite
• patch_dp.py - Update datapoint components
• patch_additional_files.py - Manage resource files

# Clone repository
git clone https://github.com/Danau5tin/tbench-agentic-data-pipeline.git
cd tbench-agentic-data-pipeline

# Install dependencies
uv sync

# Initialize seed tasks
python init_seed_tasks.py <path_to_terminal_bench_tasks>

# Launch agents with Claude Code
# Idea Agent:
"See @agents/idea_agent_workspace/workflow_instructions.md - you are the idea generation agent, go!"

# Builder Agent:
"See @agents/dp_builder_workspace/workflow_instructions.md - you are the datapoint builder agent, go!"

# Review Agent:
"See @agents/review_agent_workspace/workflow_instructions.md - you are the quality review agent, go!"

• Separation of concerns: Each agent excels at one task
• Parallel scaling: Multiple instances per type
• Quality gates: Three-stage validation ensures high standards

• Simplicity: No complex message passing
• Reliability: File operations are atomic
• Debugging: Easy to inspect intermediate states

• Coordination: Prevents duplicate work
• Recovery: Handles agent failures gracefully
• Monitoring: Real-time pipeline visibility

Built with Claude Code 🤖 - This entire multi-agent system was developed using Claude Code, demonstrating the power of AI agents building infrastructure for other AI agents.