# Pipeline Configuration Technical Guide
## Table of Contents
1. [Overview](#overview)
2. [Configuration File Structure](#configuration-file-structure)
3. [Workflow Section](#workflow-section)
4. [Defaults Section](#defaults-section)
5. [Steps Section](#steps-section)
6. [Step Types](#step-types)
7. [Prompt Templates](#prompt-templates)
8. [Token Budget Management](#token-budget-management)
9. [Claude Options](#claude-options)
10. [Advanced Features](#advanced-features)
11. [Complete Examples](#complete-examples)
12. [Best Practices](#best-practices)
13. [Troubleshooting](#troubleshooting)
## Overview
Pipeline configuration files are written in YAML format and define the complete workflow for orchestrating Gemini (planning/decision-making) and Claude (code execution) to accomplish complex tasks.
### Basic Structure
```yaml
workflow:
name: "string" # Required: Workflow identifier
checkpoint_enabled: boolean # Optional: Enable state saving
workspace_dir: "string" # Optional: Claude's sandbox directory
checkpoint_dir: "string" # Optional: Checkpoint storage location
defaults: # Optional: Default settings
gemini_model: "string" # Gemini model to use
gemini_token_budget: {} # Token limits for Gemini
claude_output_format: "string" # Output format for Claude
output_dir: "string" # Where to save outputs
gemini_functions: {} # Optional: Function definitions
steps: [] # Required: List of workflow steps
```
## Configuration File Structure
### Root Level
The configuration file must have a single root key `workflow`:
```yaml
workflow:
# All configuration goes here
```
### Required Fields
- `workflow.name`: Unique identifier for the workflow
- `workflow.steps`: Array of step definitions
### Optional Fields
- `workflow.checkpoint_enabled`: Enable workflow state saving (default: false)
- `workflow.workspace_dir`: Directory for Claude's file operations (default: "./workspace")
- `workflow.checkpoint_dir`: Where to save checkpoints (default: "./checkpoints")
- `workflow.defaults`: Default settings for all steps
## Workflow Section
The workflow section contains global settings:
```yaml
workflow:
name: "data_processor"
checkpoint_enabled: true
workspace_dir: "./workspace"
checkpoint_dir: "./checkpoints"
```
### Field Details
#### `name` (string, required)
- Identifies the workflow in logs and outputs
- Used in output directory naming
- Should be descriptive and unique
#### `checkpoint_enabled` (boolean, optional)
- When `true`, saves state after each step
- Allows resuming interrupted workflows
- Checkpoints include step results and timestamps
#### `workspace_dir` (string, optional)
- Directory where Claude performs file operations
- Created automatically if it doesn't exist
- Provides sandboxing for Claude's file access
- Default: "./workspace"
#### `checkpoint_dir` (string, optional)
- Directory for storing checkpoint files
- Created automatically if needed
- Default: "./checkpoints"
## Defaults Section
Sets default values for all steps:
```yaml
defaults:
gemini_model: "gemini-2.5-flash-lite-preview-06-17"
gemini_token_budget:
max_output_tokens: 2048
temperature: 0.7
top_p: 0.95
top_k: 40
claude_output_format: "json"
output_dir: "./outputs/my_workflow"
```
### Available Defaults
#### `gemini_model` (string)
Available models:
- `"gemini-2.5-flash"` - Fast, efficient model
- `"gemini-2.5-flash-lite-preview-06-17"` - Lightweight version
- `"gemini-2.5-pro"` - Most capable model
- `"gemini-2.0-flash"` - Previous generation
#### `gemini_token_budget` (object)
Default token limits for Gemini:
- `max_output_tokens`: Maximum response length (256-8192)
- `temperature`: Randomness control (0.0-1.0)
- `top_p`: Nucleus sampling (0.0-1.0)
- `top_k`: Top-k sampling (1-40)
#### `claude_output_format` (string)
Default output format for Claude:
- `"json"` - Structured JSON output
- `"text"` - Plain text output
- `"stream-json"` - Streaming JSON
#### `output_dir` (string)
Default directory for saving step outputs.
## Steps Section
The heart of the configuration - defines the workflow sequence:
```yaml
steps:
- name: "analyze"
type: "gemini"
role: "brain"
# ... step configuration
- name: "implement"
type: "claude"
role: "muscle"
# ... step configuration
```
### Step Fields
#### Common Fields (all step types)
##### `name` (string, required)
- Unique identifier for the step
- Referenced by other steps via `previous_response`
- Used in output filenames
##### `type` (string, required)
Options:
- `"gemini"` - Gemini AI step
- `"claude"` - Claude AI step
- `"parallel_claude"` - Multiple Claude instances
##### `role` (string, optional)
- `"brain"` - Decision-making role (typically Gemini)
- `"muscle"` - Execution role (typically Claude)
- Used for documentation/clarity
##### `condition` (string, optional)
Skip step based on previous results:
```yaml
condition: "previous_step.field_name"
```
##### `output_to_file` (string, optional)
Save step output to specified file:
```yaml
output_to_file: "analysis_result.json"
```
## Step Types
### 1. Gemini Step
For planning, analysis, and decision-making:
```yaml
- name: "plan_implementation"
type: "gemini"
role: "brain"
model: "gemini-2.5-flash" # Override default
token_budget:
max_output_tokens: 4096
temperature: 0.5
prompt:
- type: "static"
content: "Create an implementation plan..."
output_to_file: "plan.json"
```
#### Gemini-Specific Fields
##### `model` (string, optional)
Override the default Gemini model for this step.
##### `token_budget` (object, optional)
Override default token limits:
```yaml
token_budget:
max_output_tokens: 8192 # More tokens for detailed output
temperature: 0.3 # Lower for focused responses
top_p: 0.9 # Adjust nucleus sampling
top_k: 20 # Limit token choices
```
##### `functions` (array, optional)
Enable function calling:
```yaml
functions:
- "evaluate_code"
- "generate_tests"
```
### 2. Claude Step
For code execution and file manipulation:
```yaml
- name: "implement_feature"
type: "claude"
role: "muscle"
claude_options:
print: true
max_turns: 15
allowed_tools: ["Write", "Edit", "Read", "Bash"]
output_format: "json"
cwd: "./workspace"
prompt:
- type: "static"
content: "Implement the following..."
output_to_file: "implementation.json"
```
#### Claude-Specific Fields
##### `claude_options` (object, optional)
Configuration for Claude CLI:
```yaml
claude_options:
print: true # Non-interactive mode
output_format: "json" # Response format
max_turns: 20 # Conversation limit
allowed_tools: ["Write", "Edit", "Read", "Bash", "Search"]
verbose: true # Detailed logging
append_system_prompt: "Focus on clean code"
cwd: "./workspace/project" # Working directory
```
### 3. Parallel Claude Step
Run multiple Claude instances simultaneously:
```yaml
- name: "parallel_development"
type: "parallel_claude"
parallel_tasks:
- id: "backend"
claude_options:
max_turns: 15
cwd: "./workspace/backend"
prompt:
- type: "static"
content: "Implement REST API..."
output_to_file: "backend.json"
- id: "frontend"
claude_options:
max_turns: 15
cwd: "./workspace/frontend"
prompt:
- type: "static"
content: "Implement React UI..."
output_to_file: "frontend.json"
```
#### Parallel-Specific Fields
##### `parallel_tasks` (array, required)
List of task definitions, each with:
- `id`: Unique task identifier
- `claude_options`: Task-specific Claude settings
- `prompt`: Task prompt configuration
- `output_to_file`: Task output file
## Prompt Templates
Prompts are built from components that can reference files and previous outputs:
### 1. Static Content
Fixed text content:
```yaml
prompt:
- type: "static"
content: |
Analyze this requirement:
- Feature A
- Feature B
```
### 2. File Content
Load content from files:
```yaml
prompt:
- type: "file"
path: "requirements.txt"
- type: "file"
path: "src/main.py"
```
### 3. Previous Response
Reference output from earlier steps:
```yaml
prompt:
- type: "static"
content: "Based on the plan:"
- type: "previous_response"
step: "planning_step"
```
With field extraction:
```yaml
prompt:
- type: "previous_response"
step: "analysis_step"
extract: "issues" # Extract specific field from JSON
```
### 4. Complex Example
Combining all prompt types:
```yaml
prompt:
- type: "static"
content: "Project requirements:"
- type: "file"
path: "requirements.md"
- type: "static"
content: "\n\nPrevious analysis found these issues:"
- type: "previous_response"
step: "code_review"
extract: "critical_issues"
- type: "static"
content: "\n\nPlease fix all critical issues."
```
## Token Budget Management
Fine-tune AI response lengths and quality:
### Understanding Token Budgets
```yaml
token_budget:
max_output_tokens: 4096 # Response length limit
temperature: 0.7 # Randomness (0=deterministic, 1=creative)
top_p: 0.95 # Cumulative probability cutoff
top_k: 40 # Consider top K tokens
```
### Common Configurations
#### Concise Analysis
```yaml
token_budget:
max_output_tokens: 1024
temperature: 0.5
```
#### Detailed Planning
```yaml
token_budget:
max_output_tokens: 4096
temperature: 0.7
```
#### Code Generation
```yaml
token_budget:
max_output_tokens: 8192
temperature: 0.3 # Lower for consistency
```
#### Creative Writing
```yaml
token_budget:
max_output_tokens: 2048
temperature: 0.9 # Higher for variety
```
## Claude Options
### Tool Configuration
Control which tools Claude can use:
```yaml
allowed_tools: ["Write", "Edit", "Read"] # File operations only
allowed_tools: ["Write", "Edit", "Read", "Bash"] # Include shell
allowed_tools: ["Write", "Edit", "Read", "Search", "Bash"] # Full access
```
### Turn Limits
Set conversation length:
```yaml
max_turns: 5 # Quick tasks
max_turns: 15 # Standard implementation
max_turns: 30 # Complex projects
```
### Working Directory
Sandbox file operations:
```yaml
cwd: "./workspace" # Default sandbox
cwd: "./workspace/backend" # Project-specific
cwd: "/tmp/safe_dir" # Absolute path
```
## Advanced Features
### 1. Conditional Execution
Skip steps based on conditions:
```yaml
steps:
- name: "check_quality"
type: "gemini"
prompt:
- type: "static"
content: "Review code quality. Set needs_fixes=true if issues found."
- name: "fix_issues"
type: "claude"
condition: "check_quality.needs_fixes" # Only runs if true
prompt:
- type: "static"
content: "Fix the identified issues"
```
### 2. Function Calling
Define functions for Gemini:
```yaml
workflow:
gemini_functions:
evaluate_code:
description: "Evaluate code quality and security"
parameters:
type: object
properties:
quality_score:
type: integer
description: "Score from 1-10"
security_issues:
type: array
items:
type: string
needs_refactoring:
type: boolean
required: ["quality_score", "security_issues"]
steps:
- name: "code_review"
type: "gemini"
functions:
- "evaluate_code"
prompt:
- type: "static"
content: "Review this code and call evaluate_code function"
```
### 3. Dynamic Workflows
Build prompts from multiple sources:
```yaml
steps:
- name: "gather_context"
type: "gemini"
prompt:
- type: "file"
path: "context/project.md"
- type: "file"
path: "context/architecture.md"
- type: "static"
content: "Summarize the project context"
- name: "generate_tasks"
type: "gemini"
prompt:
- type: "previous_response"
step: "gather_context"
- type: "static"
content: "Based on this context, generate implementation tasks"
- name: "implement"
type: "claude"
prompt:
- type: "previous_response"
step: "generate_tasks"
- type: "static"
content: "Implement these tasks"
```
## Complete Examples
### Example 1: Code Review and Fix
```yaml
workflow:
name: "code_reviewer"
workspace_dir: "./workspace"
defaults:
gemini_model: "gemini-2.5-flash"
output_dir: "./outputs/review"
steps:
- name: "analyze_code"
type: "gemini"
token_budget:
max_output_tokens: 2048
temperature: 0.3
prompt:
- type: "static"
content: "Review this code for bugs, security issues, and style:"
- type: "file"
path: "src/main.py"
output_to_file: "analysis.json"
- name: "fix_issues"
type: "claude"
claude_options:
max_turns: 10
allowed_tools: ["Read", "Edit"]
cwd: "./workspace"
prompt:
- type: "static"
content: "Fix all issues identified in the review:"
- type: "previous_response"
step: "analyze_code"
output_to_file: "fixes.json"
```
### Example 2: Full Application Development
```yaml
workflow:
name: "app_builder"
checkpoint_enabled: true
workspace_dir: "./workspace"
defaults:
gemini_model: "gemini-2.5-flash"
gemini_token_budget:
max_output_tokens: 4096
temperature: 0.7
claude_output_format: "json"
output_dir: "./outputs/app"
gemini_functions:
design_architecture:
description: "Design application architecture"
parameters:
type: object
properties:
components:
type: array
items:
type: object
properties:
name:
type: string
type:
type: string
enum: ["frontend", "backend", "database", "service"]
dependencies:
type: array
items:
type: string
steps:
- name: "design"
type: "gemini"
role: "brain"
functions:
- "design_architecture"
prompt:
- type: "file"
path: "requirements.md"
- type: "static"
content: |
Design a microservices architecture for these requirements.
Call design_architecture with the component structure.
output_to_file: "architecture.json"
- name: "plan_implementation"
type: "gemini"
role: "brain"
token_budget:
max_output_tokens: 8192
prompt:
- type: "static"
content: "Create detailed implementation instructions for each component:"
- type: "previous_response"
step: "design"
output_to_file: "implementation_plan.json"
- name: "implement_services"
type: "parallel_claude"
parallel_tasks:
- id: "auth_service"
claude_options:
max_turns: 20
allowed_tools: ["Write", "Edit", "Read", "Bash"]
cwd: "./workspace/services/auth"
prompt:
- type: "static"
content: "Implement the authentication service based on the plan:"
- type: "previous_response"
step: "plan_implementation"
extract: "auth_service"
output_to_file: "auth_implementation.json"
- id: "api_gateway"
claude_options:
max_turns: 20
allowed_tools: ["Write", "Edit", "Read", "Bash"]
cwd: "./workspace/services/gateway"
prompt:
- type: "static"
content: "Implement the API gateway based on the plan:"
- type: "previous_response"
step: "plan_implementation"
extract: "api_gateway"
output_to_file: "gateway_implementation.json"
- id: "frontend"
claude_options:
max_turns: 25
allowed_tools: ["Write", "Edit", "Read", "Bash"]
cwd: "./workspace/frontend"
prompt:
- type: "static"
content: "Implement the React frontend based on the plan:"
- type: "previous_response"
step: "plan_implementation"
extract: "frontend"
output_to_file: "frontend_implementation.json"
- name: "integration_tests"
type: "gemini"
role: "brain"
prompt:
- type: "static"
content: "Design integration tests for the implemented services:"
- type: "previous_response"
step: "implement_services"
output_to_file: "test_plan.json"
- name: "implement_tests"
type: "claude"
role: "muscle"
claude_options:
max_turns: 15
allowed_tools: ["Write", "Read", "Bash"]
cwd: "./workspace/tests"
prompt:
- type: "previous_response"
step: "integration_tests"
- type: "static"
content: "Implement these integration tests"
output_to_file: "tests_implementation.json"
- name: "final_review"
type: "gemini"
role: "brain"
token_budget:
max_output_tokens: 2048
temperature: 0.3
prompt:
- type: "static"
content: |
Review the complete implementation:
1. Check if all requirements are met
2. Identify any missing features
3. Suggest improvements
Set 'ready_for_deployment' to true if acceptable.
- type: "previous_response"
step: "implement_services"
output_to_file: "final_review.json"
```
## Best Practices
### 1. Token Budget Optimization
- Start with conservative limits and increase as needed
- Use lower temperatures (0.1-0.3) for technical tasks
- Use higher temperatures (0.7-0.9) for creative tasks
- Monitor token usage in debug logs
### 2. Step Design
- Keep prompts focused on single responsibilities
- Use descriptive step names for easy reference
- Save outputs for all important steps
- Build complex prompts from simpler components
### 3. Error Handling
- Set appropriate `max_turns` for Claude tasks
- Use conditions to handle failure cases
- Include validation steps after implementation
- Review debug logs when issues occur
### 4. Workspace Management
- Use subdirectories for different components
- Set appropriate `cwd` in claude_options
- Keep workspace separate from project files
- Clean workspace between runs if needed
### 5. Prompt Engineering
- Be specific in instructions
- Provide examples when possible
- Use previous responses to maintain context
- Break complex tasks into smaller steps
## Troubleshooting
### Common Issues
#### 1. Claude Hits Turn Limit
```yaml
# Increase max_turns
claude_options:
max_turns: 30 # Increase from default
```
#### 2. Gemini Response Truncated
```yaml
# Increase token limit
token_budget:
max_output_tokens: 8192 # Maximum allowed
```
#### 3. Files Created in Wrong Location
```yaml
# Check workspace configuration
workflow:
workspace_dir: "./workspace"
steps:
- name: "implement"
claude_options:
cwd: "./workspace/project" # Must be under workspace_dir
```
#### 4. Step References Not Found
```yaml
# Ensure step names match exactly
- name: "step_one" # This exact name
type: "gemini"
- name: "step_two"
prompt:
- type: "previous_response"
step: "step_one" # Must match exactly
```
#### 5. JSON Parsing Errors
```yaml
# Use appropriate output format
claude_options:
output_format: "json" # For structured data
# or
output_format: "text" # For plain text
```
### Debug Techniques
1. **Check Debug Logs**
```bash
python view_debug.py -l
```
2. **Verify File Creation**
```bash
python view_debug.py -w
```
3. **Test Individual Steps**
- Comment out later steps
- Run pipeline partially
- Check intermediate outputs
4. **Validate YAML Syntax**
```bash
python -c "import yaml; yaml.safe_load(open('config.yaml'))"
```
## Schema Reference
### Complete Configuration Schema
```yaml
workflow:
name: string # Required
checkpoint_enabled: boolean # Optional
workspace_dir: string # Optional
checkpoint_dir: string # Optional
# NEW: Enhanced Claude authentication and environment configuration
claude_auth: # Optional
auto_check: boolean # Verify auth before starting
provider: enum["anthropic", "aws_bedrock", "google_vertex"]
fallback_mock: boolean # Use mocks if auth fails in dev
diagnostics: boolean # Run AuthChecker diagnostics
environment: # Optional
mode: enum["development", "production", "test"]
debug_level: enum["basic", "detailed", "performance"]
cost_alerts:
enabled: boolean
threshold_usd: float
notify_on_exceed: boolean
defaults: # Optional
gemini_model: string
gemini_token_budget:
max_output_tokens: integer
temperature: float
top_p: float
top_k: integer
claude_output_format: enum["json", "text", "stream-json"]
claude_preset: enum["development", "production", "analysis", "chat"] # NEW
output_dir: string
gemini_functions: # Optional
function_name:
description: string
parameters: object # JSON Schema
steps: # Required
- name: string # Required
type: enum["gemini", "claude", "parallel_claude", "claude_session", "claude_smart", "claude_extract", "claude_batch", "claude_robust"] # Enhanced
role: enum["brain", "muscle"] # Optional
condition: string # Optional
output_to_file: string # Optional
# For type: "gemini"
model: string # Optional
token_budget: object # Optional
functions: array[string] # Optional
# For type: "claude" (Enhanced options)
claude_options: # Optional
# Core Configuration
max_turns: integer
output_format: enum["text", "json", "stream_json"]
verbose: boolean
# Tool Management
allowed_tools: array[string]
disallowed_tools: array[string]
# System Prompts
system_prompt: string
append_system_prompt: string
# Working Environment
cwd: string
# Permission Management (Future: MCP Support)
permission_mode: enum["default", "accept_edits", "bypass_permissions", "plan"]
permission_prompt_tool: string
# Advanced Features (Future)
mcp_config: string
# Session Management
session_id: string
resume_session: boolean
# Performance & Reliability
retry_config:
max_retries: integer
backoff_strategy: enum["linear", "exponential"]
retry_on: array[string]
timeout_ms: integer
# Debug & Monitoring
debug_mode: boolean
telemetry_enabled: boolean
cost_tracking: boolean
# NEW: For type: "claude_smart" (Using OptionBuilder presets)
preset: enum["development", "production", "analysis", "chat"]
environment_aware: boolean
# NEW: For type: "claude_extract" (Enhanced content processing)
extraction_config:
use_content_extractor: boolean
format: enum["text", "json", "structured", "summary", "markdown"]
post_processing: array[string]
max_summary_length: integer
include_metadata: boolean
# NEW: For type: "claude_session" (Session management)
session_config:
persist: boolean
session_name: string
continue_on_restart: boolean
checkpoint_frequency: integer
description: string
# NEW: For type: "claude_batch" (Batch processing)
batch_config:
max_parallel: integer
timeout_per_task: integer
consolidate_results: boolean
tasks: array
- file: string
prompt: string
# NEW: For type: "claude_robust" (Error recovery)
retry_config:
max_retries: integer
backoff_strategy: enum["linear", "exponential"]
retry_conditions: array[string]
fallback_action: string
# For type: "parallel_claude"
parallel_tasks: array # Required
- id: string
claude_options: object
prompt: array
output_to_file: string
prompt: array # Required (except parallel_claude)
- type: enum["static", "file", "previous_response", "session_context", "claude_continue"] # Enhanced
# For type: "static"
content: string
# For type: "file"
path: string
# For type: "previous_response"
step: string
extract: string # Optional
extract_with: enum["content_extractor"] # NEW: Use ContentExtractor
summary: boolean # NEW: Summarize content
max_length: integer # NEW: Limit extracted content length
# NEW: For type: "session_context"
session_id: string
include_last_n: integer
# NEW: For type: "claude_continue"
new_prompt: string
```
### Enhanced Claude Options Reference
The enhanced `claude_options` section now supports the full feature set of the Claude Code SDK:
#### Basic Configuration
- `max_turns`: Maximum conversation turns (integer)
- `output_format`: Response format - "text", "json", or "stream_json"
- `verbose`: Enable detailed logging (boolean)
#### Tool Management
- `allowed_tools`: List of permitted tool names (array of strings)
- `disallowed_tools`: List of explicitly forbidden tools (array of strings)
#### System Prompts
- `system_prompt`: Custom system prompt override (string)
- `append_system_prompt`: Additional system prompt to append (string)
#### Working Environment
- `cwd`: Working directory for Claude operations (string)
#### Session Management
- `session_id`: Explicit session identifier for continuation (string)
- `resume_session`: Automatically resume existing session if available (boolean)
#### Performance & Reliability
- `retry_config`: Retry mechanism configuration (object)
- `max_retries`: Maximum number of retry attempts (integer)
- `backoff_strategy`: "linear" or "exponential" backoff (string)
- `retry_on`: List of conditions that trigger retries (array)
- `timeout_ms`: Request timeout in milliseconds (integer)
#### Monitoring & Debug
- `debug_mode`: Enable debug output and diagnostics (boolean)
- `telemetry_enabled`: Enable performance telemetry (boolean)
- `cost_tracking`: Track and report API costs (boolean)
### Smart Configuration Presets
The new `claude_smart` step type supports OptionBuilder presets:
- **`development`**: Permissive settings, verbose logging, full tool access
- **`production`**: Restricted settings, minimal tools, safe defaults
- **`analysis`**: Read-only tools, optimized for code analysis
- **`chat`**: Simple conversation settings, basic tools
### Future-Ready Features
The schema includes placeholders for planned features:
#### MCP Integration
- `mcp_config`: Path to MCP server configuration file
- `permission_prompt_tool`: Tool for handling permission prompts
- `permission_mode`: Permission handling strategy
#### Advanced Content Processing
- Content extraction with multiple format options
- Post-processing pipeline support
- Metadata inclusion and summarization
This guide provides comprehensive documentation for creating and managing pipeline configuration files. Use it as a reference when building your own AI-orchestrated workflows.
