# Recursive Pipelines Guide
## Table of Contents
1. [Overview](#overview)
2. [Core Concepts](#core-concepts)
3. [Basic Usage](#basic-usage)
4. [Context Management](#context-management)
5. [Safety Features](#safety-features)
6. [Developer Experience](#developer-experience)
7. [Advanced Patterns](#advanced-patterns)
8. [Performance Considerations](#performance-considerations)
9. [Best Practices](#best-practices)
10. [Examples and Use Cases](#examples-and-use-cases)
11. [Troubleshooting](#troubleshooting)
12. [API Reference](#api-reference)
13. [Related Guides](#related-guides)
## Overview
Recursive pipelines enable pipeline composition in pipeline_ex, where a pipeline step can execute another complete pipeline. This transforms pipeline_ex from a linear workflow engine into a compositional framework supporting complex, hierarchical AI engineering patterns.
### Key Benefits
- **Modularity**: Build complex workflows from simpler, tested components
- **Reusability**: Share common pipeline patterns across projects
- **Maintainability**: Update shared pipelines in one place
- **Scalability**: Compose arbitrarily complex workflows
- **Testing**: Test pipeline components in isolation
### Architecture
```mermaid
graph TD
A[Main Pipeline] --> B[Executor]
B --> C{Step Type?}
C -->|pipeline| D[NestedPipeline Module]
C -->|claude| E[Claude Module]
C -->|gemini| F[Gemini Module]
D --> G[Load Sub-Pipeline]
G --> H[Create Nested Context]
H --> I[Execute Sub-Pipeline]
I --> J[Extract Results]
J --> K[Merge to Parent Context]
style D fill:#f9f,stroke:#333,stroke-width:4px
```
## Core Concepts
### Pipeline Composition
Recursive pipelines allow you to nest complete pipelines within other pipelines:
```yaml
# Instead of monolithic 50+ step pipelines
workflow:
name: "complete_analysis"
steps:
# Compose from reusable components
- name: "extract_data"
type: "pipeline"
pipeline_file: "./pipelines/data_extraction.yaml"
- name: "clean_data"
type: "pipeline"
pipeline_file: "./pipelines/data_cleaning.yaml"
- name: "analyze"
type: "pipeline"
pipeline_file: "./pipelines/analysis.yaml"
```
### Context Isolation
Each nested pipeline runs in its own isolated context:
- Separate workspace directories
- Independent checkpoint management
- Isolated error handling
- Controlled variable passing
### Nesting Depth Tracking
The system tracks nesting depth to prevent infinite recursion and resource exhaustion.
## Basic Usage
### Step Definition Schema
```yaml
- name: "sub_workflow"
type: "pipeline"
# Pipeline source (one required)
pipeline_file: "./path/to/pipeline.yaml" # External file
pipeline_ref: "registered_pipeline_name" # From registry (future)
pipeline: # Inline definition
name: "inline_pipeline"
steps:
- name: "step1"
type: "claude"
prompt: "..."
# Input mapping (optional)
inputs:
data: "{{steps.previous.result}}"
config: "{{workflow.settings}}"
mode: "production" # Static values
# Output extraction (optional)
outputs:
- "final" # Simple extraction
- path: "analysis.metrics.accuracy"
as: "accuracy_score" # With aliasing
# Execution configuration (optional)
config:
inherit_context: true
workspace_dir: "./nested/${step.name}"
timeout_seconds: 300
max_depth: 5
```
### Simple Example
```yaml
workflow:
name: "data_processor"
steps:
- name: "prepare"
type: "pipeline"
pipeline:
name: "data_prep"
steps:
- name: "validate"
type: "claude"
prompt: "Validate this data format"
- name: "clean"
type: "claude"
prompt: "Clean and normalize the data"
- name: "analyze"
type: "pipeline"
pipeline_file: "./analysis_pipeline.yaml"
```
## Context Management
### Context Inheritance
Control how much context is shared between parent and child pipelines:
```yaml
- name: "nested_step"
type: "pipeline"
pipeline_file: "./child.yaml"
config:
inherit_context: true # Pass parent context vars
inherit_providers: true # Use parent provider configs
inherit_functions: false # Don't inherit function defs
```
When `inherit_context` is true, the child pipeline can access:
- Parent's global variables
- Parent's function definitions
- Parent's provider configurations
- Parent's workflow metadata
### Input Mapping
Pass specific data from parent to child with full template resolution:
```yaml
- name: "process_data"
type: "pipeline"
pipeline_file: "./processor.yaml"
inputs:
# Variable references from previous steps
source_data: "{{steps.extract.result}}"
nested_field: "{{steps.prepare.result.data.items}}"
# Global variables
api_key: "{{global_vars.api_key}}"
# Workflow configuration
user_config: "{{workflow.user_settings}}"
# Static values
processing_mode: "batch"
max_items: 1000
# Complex expressions (future)
threshold: "{{multiply(workflow.base_threshold, 1.5)}}"
```
**Type Preservation**: Single template references preserve their original type:
- `"{{steps.data.count}}"` → `42` (integer)
- `"{{steps.data.result}}"` → `{"name": "test", "value": 100}` (object)
- `"Count: {{steps.data.count}}"` → `"Count: 42"` (string)
### Output Extraction
Extract specific results from nested pipelines with flexible patterns:
```yaml
outputs:
# Simple extraction - gets steps.stepname.result
- "final_result"
# Path-based extraction with deep access
- path: "analysis.metrics.accuracy"
as: "accuracy_score"
# Multiple path-based extractions
- path: "step1.data"
as: "processed_data"
- path: "step2.summary.text"
as: "summary"
# Extract entire step results
- "validation"
- "transformation"
```
Example usage:
```yaml
- name: "analyze"
type: "pipeline"
pipeline_file: "./analysis.yaml"
outputs:
- path: "metrics.performance.score"
as: "perf_score"
- path: "report.summary"
as: "summary_text"
- name: "use_results"
type: "claude"
prompt: |
Performance score: {{steps.analyze.perf_score}}
Summary: {{steps.analyze.summary_text}}
```
### Variable Resolution
The custom template resolution engine supports pipeline-specific patterns:
1. **Step Results**: `{{steps.stepname.result}}` or `{{steps.stepname.result.field.nested}}`
2. **Global Variables**: `{{global_vars.variable_name}}`
3. **Workflow Data**: `{{workflow.config.setting}}`
4. **Direct Variables**: `{{variable_name}}`
Resolution order:
1. Check current (child) context first
2. If not found and inheritance enabled, check parent context
3. Continue up the context chain
4. Return the original template if variable not found (graceful fallback)
## Safety Features
The recursive pipeline system includes comprehensive safety mechanisms to prevent common failure modes:
### Recursion Protection
#### Depth Limiting
Control maximum nesting depth to prevent stack overflow:
```yaml
- name: "nested_pipeline"
type: "pipeline"
pipeline_file: "./child.yaml"
config:
max_depth: 5 # Override default limit
```
Environment-specific defaults:
```elixir
# config/dev.exs - Development
config :pipeline,
max_nesting_depth: 15, # More permissive for experimentation
max_total_steps: 2000
# config/test.exs - Testing
config :pipeline,
max_nesting_depth: 5, # Restricted for fast tests
max_total_steps: 100
# config/prod.exs - Production
config :pipeline,
max_nesting_depth: 8, # Conservative for stability
max_total_steps: 500
```
#### Circular Dependency Detection
The system automatically detects and prevents circular dependencies:
```yaml
# pipeline_a.yaml
- name: "call_b"
type: "pipeline"
pipeline_file: "./pipeline_b.yaml"
# pipeline_b.yaml
- name: "call_a"
type: "pipeline"
pipeline_file: "./pipeline_a.yaml" # ERROR: Circular dependency!
```
Error message:
```
Error: Circular dependency detected in pipeline execution chain:
main_workflow -> pipeline_a -> pipeline_b -> pipeline_a
Execution chain:
1. main_workflow (depth: 0)
2. pipeline_a (depth: 1)
3. pipeline_b (depth: 2)
4. pipeline_a (depth: 3) <- Circular reference
```
### Resource Management
#### Memory Monitoring
Real-time memory usage tracking with configurable limits:
```yaml
config:
memory_limit_mb: 1024 # 1GB limit
memory_warning_mb: 768 # Warning at 75% usage
```
Memory pressure warnings:
```
Warning: Memory usage approaching limit
Current: 850MB / 1024MB (83%)
Pipeline: data_processor (depth: 2)
Consider optimizing data handling or increasing limit
```
#### Timeout Protection
Prevent runaway pipelines with execution time limits:
```yaml
config:
timeout_seconds: 300 # 5 minute timeout
max_retries: 2 # Retry attempts on timeout
```
#### Step Count Limits
Prevent exponential pipeline expansion:
```yaml
config:
max_total_steps: 1000 # Total steps across all nested pipelines
```
### Workspace Isolation
Each nested pipeline gets its own isolated workspace:
```yaml
config:
workspace_dir: "./nested/${step.name}" # Isolated directory
cleanup_on_error: true # Auto cleanup
cleanup_on_success: true # Clean after completion
```
### Resource Cleanup
Automatic cleanup on all execution paths:
- **Workspace directories**: Removed after pipeline completion
- **Memory structures**: Large data cleared from context
- **Context chains**: Parent references cleaned to prevent leaks
- **Temporary files**: All temp files removed
## Developer Experience
The recursive pipeline system includes comprehensive developer tools for debugging, monitoring, and optimization:
### Enhanced Error Messages
Production-grade error reporting with full execution context:
```
Pipeline execution failed in nested pipeline:
Main Pipeline: data_analysis_workflow
└─ Step: process_data (pipeline)
└─ Nested Pipeline: data_processor
└─ Step: validate_schema (claude)
└─ Error: Invalid JSON in response
Execution Stack:
1. data_analysis_workflow.process_data (depth: 0)
2. data_processor.validate_schema (depth: 1)
Context at failure:
- Total steps executed: 12
- Nesting depth: 1
- Elapsed time: 4.2s
- Memory usage: 234MB
Debug Information:
- Trace ID: a1b2c3d4-e5f6-7890-abcd-ef1234567890
- Step config: {"type": "claude", "prompt": "..."}
- Parent pipeline: data_analysis_workflow
```
### Execution Tracing
Span-based tracing system for performance analysis:
```elixir
# Enable tracing for a pipeline
- name: "complex_workflow"
type: "pipeline"
pipeline_file: "./workflow.yaml"
config:
enable_tracing: true
trace_metadata:
request_id: "{{inputs.request_id}}"
user_id: "{{inputs.user_id}}"
```
Trace output includes:
- Hierarchical span relationships
- Execution duration per span
- Resource usage metrics
- Success/failure status
- Custom metadata
### Visual Debugging
Interactive execution tree visualization:
```
Execution Tree for trace_id: a1b2c3d4
├─ main_pipeline (289.5ms) ✓
│ ├─ prepare_data (45.2ms) ✓
│ │ └─ validation: {"status": "valid", "records": 1000}
│ ├─ analysis_pipeline (198.3ms) ✓
│ │ ├─ security_scan (89.1ms) ✓
│ │ │ └─ vulnerabilities: []
│ │ ├─ performance_check (72.4ms) ✓
│ │ │ └─ bottlenecks: ["database_query_optimization"]
│ │ └─ code_quality (36.8ms) ✓
│ │ └─ score: 8.5
│ └─ generate_report (46.0ms) ✓
│ └─ report_url: "https://reports.example.com/abc123"
Performance Summary:
- Total Duration: 289.5ms
- Deepest Nesting: 2 levels
- Total Steps: 7
- Success Rate: 100%
- Average Step Duration: 41.4ms
```
### Performance Metrics
Detailed performance analysis per nesting level:
```elixir
# Performance metrics collection
Pipeline.Metrics.NestedPerformance.analyze_performance(execution_id)
# Returns:
%{
overall_metrics: %{
total_duration_ms: 289.5,
total_steps: 7,
success_rate: 1.0,
max_depth: 2
},
depth_metrics: %{
0 => %{steps: 3, avg_duration_ms: 96.5, success_rate: 1.0},
1 => %{steps: 4, avg_duration_ms: 48.6, success_rate: 1.0}
},
bottlenecks: [
%{pipeline: "analysis_pipeline", step: "security_scan", duration_ms: 89.1}
],
performance_grade: :excellent, # :excellent | :good | :fair | :poor
recommendations: [
"Consider caching security scan results for repeated analyses",
"Database query in performance_check could be optimized"
]
}
```
### Debug Commands
Interactive debugging interface:
```elixir
# Start debug session
Pipeline.Debug.NestedExecution.start_debug_session(trace_context)
# Available commands:
# - tree: Show execution tree
# - inspect <step>: Examine step details
# - search <pattern>: Search execution traces
# - analyze: Performance analysis
# - report: Generate debug report
```
### Telemetry Integration
Built-in telemetry events for monitoring:
```elixir
# Attach to telemetry events
:telemetry.attach_many(
"pipeline-monitoring",
[
[:pipeline, :nested, :start],
[:pipeline, :nested, :stop],
[:pipeline, :nested, :error],
[:pipeline, :span, :start],
[:pipeline, :span, :stop]
],
&handle_pipeline_event/4,
nil
)
# Event metadata includes:
# - pipeline_id, depth, parent_span
# - duration, memory_usage, step_count
# - error details (for error events)
```
## Advanced Patterns
### Progressive Enhancement
Build complexity gradually through connected prompts:
```yaml
workflow:
name: "progressive_analysis"
steps:
- name: "basic_scan"
type: "pipeline"
pipeline_file: "./basic_analysis.yaml"
- name: "detailed_analysis"
type: "pipeline"
pipeline_file: "./detailed_analysis.yaml"
inputs:
findings: "{{steps.basic_scan.concerns}}"
source_data: "{{workflow.input_data}}"
- name: "comprehensive_report"
type: "pipeline"
pipeline_file: "./report_generator.yaml"
inputs:
basic_results: "{{steps.basic_scan.result}}"
detailed_results: "{{steps.detailed_analysis.result}}"
```
### Conditional Sub-Pipelines
Execute different pipelines based on conditions:
```yaml
workflow:
name: "adaptive_processing"
steps:
- name: "detect_type"
type: "claude"
prompt: "Detect the data type: {{input}}"
- name: "process_text"
type: "pipeline"
pipeline_file: "./text_processor.yaml"
condition: "{{steps.detect_type.result.type == 'text'}}"
- name: "process_structured"
type: "pipeline"
pipeline_file: "./structured_processor.yaml"
condition: "{{steps.detect_type.result.type == 'structured'}}"
```
### Parallel Nested Execution
Run multiple sub-pipelines simultaneously:
```yaml
- name: "parallel_analysis"
type: "parallel_claude"
parallel_tasks:
- id: "security_scan"
type: "pipeline"
pipeline_file: "./security_analysis.yaml"
- id: "performance_scan"
type: "pipeline"
pipeline_file: "./performance_analysis.yaml"
- id: "quality_scan"
type: "pipeline"
pipeline_file: "./quality_analysis.yaml"
```
### Self-Referential Pipelines
Pipelines that can call themselves (with proper depth limits):
```yaml
workflow:
name: "recursive_analyzer"
steps:
- name: "analyze_item"
type: "claude"
prompt: "Analyze: {{inputs.item}}"
- name: "process_children"
type: "for_loop"
over: "{{steps.analyze_item.result.sub_items}}"
as: "child"
steps:
- name: "recurse"
type: "pipeline"
pipeline_ref: "recursive_analyzer" # Self-reference
inputs:
item: "{{child}}"
config:
max_depth: 3 # Prevent infinite recursion
```
### Error Recovery Pattern
Handle failures gracefully with fallback pipelines:
```yaml
workflow:
name: "resilient_processor"
steps:
- name: "primary_processing"
type: "pipeline"
pipeline_file: "./primary.yaml"
config:
continue_on_error: true
- name: "fallback_processing"
type: "pipeline"
pipeline_file: "./fallback.yaml"
condition: "{{steps.primary_processing.error != null}}"
inputs:
error: "{{steps.primary_processing.error}}"
original_input: "{{inputs.data}}"
```
## Performance Considerations
### Pipeline Caching
The system caches loaded pipelines for performance:
```elixir
# Automatic caching with TTL
pipeline_cache_ttl: 3600 # 1 hour
```
### Optimization Strategies
1. **Lazy Loading**: Sub-pipelines are loaded only when needed
2. **Context Pooling**: Reuse context structures where safe
3. **Result Streaming**: Stream large results between pipelines
4. **Parallel Execution**: Run independent sub-pipelines concurrently
### Performance Metrics
Track execution metrics:
```yaml
# Prometheus metrics example
pipeline_nested_executions_total{pipeline="analysis", depth="2"} 42
pipeline_nested_duration_seconds{pipeline="analysis", depth="2"} 1.234
pipeline_nested_memory_bytes{pipeline="analysis"} 104857600
```
## Best Practices
### Organization
```
project/
├── pipelines/
│ ├── components/ # Reusable sub-pipelines
│ │ ├── data_prep.yaml
│ │ ├── analysis.yaml
│ │ └── reporting.yaml
│ ├── workflows/ # Main orchestration pipelines
│ │ ├── full_analysis.yaml
│ │ └── quick_scan.yaml
│ └── templates/ # Pipeline templates
└── shared/
└── common_steps.yaml
```
### Design Principles
1. **Single Responsibility**: Each sub-pipeline should have one clear purpose
2. **Loose Coupling**: Minimize dependencies between pipelines
3. **Clear Interfaces**: Well-defined inputs and outputs
4. **Error Boundaries**: Proper error handling at each level
### Naming Conventions
- **Descriptive names**: `security_vulnerability_scan.yaml` not `scan.yaml`
- **Action-oriented**: `generate_api_tests.yaml` not `api_tests.yaml`
- **Domain prefixes**: `frontend_analysis.yaml`, `backend_review.yaml`
- **Version suffixes**: `data_processor_v2.yaml` for major updates
### Configuration Management
```yaml
# Use consistent configuration patterns
config:
inherit_context: true
workspace_dir: "./nested/${step.name}"
timeout_seconds: 300
max_depth: 5
```
## Examples and Use Cases
### Use Case 1: Multi-Stage Data Processing
```yaml
workflow:
name: "data_pipeline"
steps:
- name: "ingestion"
type: "pipeline"
pipeline_file: "./stages/data_ingestion.yaml"
inputs:
source: "{{inputs.data_source}}"
format: "{{inputs.format}}"
outputs:
- path: "raw_data"
as: "ingested_data"
- "metadata"
- name: "validation"
type: "pipeline"
pipeline_file: "./stages/data_validation.yaml"
inputs:
data: "{{steps.ingestion.ingested_data}}"
schema: "{{steps.ingestion.metadata.schema}}"
outputs:
- "validated_data"
- "validation_report"
- name: "transformation"
type: "pipeline"
pipeline_file: "./stages/data_transformation.yaml"
inputs:
data: "{{steps.validation.validated_data}}"
rules: "{{workflow.transformation_rules}}"
outputs:
- "processed_data"
```
### Use Case 2: Code Generation Workflow
```yaml
workflow:
name: "app_generator"
steps:
- name: "analyze_requirements"
type: "pipeline"
pipeline_file: "./generators/requirements_analysis.yaml"
inputs:
specs: "{{inputs.requirements}}"
- name: "generate_backend"
type: "pipeline"
pipeline_file: "./generators/backend_generator.yaml"
inputs:
architecture: "{{steps.analyze_requirements.backend_spec}}"
database: "{{steps.analyze_requirements.db_requirements}}"
- name: "generate_frontend"
type: "pipeline"
pipeline_file: "./generators/frontend_generator.yaml"
inputs:
ui_spec: "{{steps.analyze_requirements.ui_requirements}}"
api_spec: "{{steps.generate_backend.api_definition}}"
- name: "integration_tests"
type: "pipeline"
pipeline_file: "./generators/test_generator.yaml"
inputs:
backend_code: "{{steps.generate_backend.result}}"
frontend_code: "{{steps.generate_frontend.result}}"
```
### Use Case 3: Comprehensive Analysis Framework
```yaml
workflow:
name: "security_audit"
steps:
- name: "static_analysis"
type: "parallel_claude"
parallel_tasks:
- id: "code_security"
type: "pipeline"
pipeline_file: "./security/code_analysis.yaml"
- id: "dependency_check"
type: "pipeline"
pipeline_file: "./security/dependency_audit.yaml"
- id: "config_review"
type: "pipeline"
pipeline_file: "./security/config_analysis.yaml"
- name: "dynamic_analysis"
type: "pipeline"
pipeline_file: "./security/runtime_analysis.yaml"
inputs:
static_findings: "{{steps.static_analysis.result}}"
- name: "compliance_check"
type: "pipeline"
pipeline_file: "./security/compliance_review.yaml"
inputs:
security_findings: "{{steps.dynamic_analysis.vulnerabilities}}"
static_results: "{{steps.static_analysis.result}}"
- name: "final_report"
type: "pipeline"
pipeline_file: "./reporting/security_report.yaml"
inputs:
all_findings: "{{steps.compliance_check.consolidated_results}}"
```
## Troubleshooting
### Common Issues
#### 1. Circular Dependency Error
```
Error: Circular dependency detected: pipeline_a -> pipeline_b -> pipeline_a
```
**Solution**: Review your pipeline call chain and eliminate cycles.
#### 2. Maximum Depth Exceeded
```
Error: Maximum nesting depth (10) exceeded
```
**Solutions**:
- Reduce nesting levels
- Increase `max_depth` configuration
- Refactor deep hierarchies into flatter structures
#### 3. Pipeline Not Found
```
Error: Pipeline file not found: ./missing_pipeline.yaml
```
**Solutions**:
- Verify file path is correct
- Check file permissions
- Ensure file exists in expected location
#### 4. Memory Limit Exceeded
```
Error: Memory limit exceeded: 1500MB > 1024MB
```
**Solutions**:
- Increase memory limit in configuration
- Optimize large data handling
- Stream results instead of loading all in memory
### Debugging Tools
#### Execution Tree Visualization
```bash
# Enable debug mode for detailed execution logging
PIPELINE_DEBUG=true mix pipeline.run complex_workflow.yaml
```
Debug output shows execution hierarchy:
```
Execution Tree:
├─ main_pipeline (depth: 0)
│ ├─ data_prep (depth: 1)
│ │ └─ validation: success
│ ├─ analysis (depth: 1)
│ │ ├─ security_scan (depth: 2)
│ │ └─ performance_check (depth: 2)
│ └─ reporting (depth: 1)
```
#### Enhanced Error Messages
Errors include full execution context:
```
Pipeline execution failed in nested pipeline:
Main Pipeline: data_processor
└─ Step: analyze_data (pipeline)
└─ Nested Pipeline: analysis_pipeline
└─ Step: extract_themes (claude)
└─ Error: API timeout after 30s
Execution Stack:
1. data_processor.analyze_data (depth: 0)
2. analysis_pipeline.extract_themes (depth: 1)
Context at failure:
- Total steps executed: 15
- Nesting depth: 1
- Elapsed time: 32.5s
```
### Performance Debugging
Monitor performance metrics:
```elixir
# Enable telemetry events
:telemetry.attach_many(
"pipeline-metrics",
[
[:pipeline, :nested, :execution],
[:pipeline, :span, :start],
[:pipeline, :span, :stop]
],
&handle_telemetry_event/4,
nil
)
```
## API Reference
### Step Configuration
```yaml
type: pipeline
name: string # Step name (required)
pipeline_file: string # Path to external pipeline file
pipeline_ref: string # Reference to registered pipeline (future)
pipeline: object # Inline pipeline definition
inputs: map # Input variable mappings
outputs: array # Output extraction configuration
config: # Execution configuration
inherit_context: boolean # Inherit parent context (default: false)
inherit_providers: boolean # Inherit provider configs (default: true)
inherit_functions: boolean # Inherit function definitions (default: false)
workspace_dir: string # Nested workspace directory
checkpoint_enabled: boolean # Enable checkpointing (default: false)
timeout_seconds: integer # Execution timeout
max_retries: integer # Retry attempts
continue_on_error: boolean # Continue on sub-pipeline error
max_depth: integer # Maximum nesting depth
memory_limit_mb: integer # Memory limit in MB
enable_tracing: boolean # Enable execution tracing
```
### Elixir API
```elixir
# Execute nested pipeline
Pipeline.Step.NestedPipeline.execute(step_config, context)
# Context management
Pipeline.Context.Nested.create_nested_context(parent_context, step_config)
Pipeline.Context.Nested.extract_outputs(results, output_config)
Pipeline.Context.Nested.resolve_template(template, context)
# Safety checks
Pipeline.Safety.RecursionGuard.check_all_safety(pipeline_id, context, limits)
Pipeline.Safety.ResourceMonitor.check_limits(usage, limits)
Pipeline.Safety.SafetyManager.check_safety(pipeline_id, context, config)
# Error handling
Pipeline.Error.NestedPipeline.format_nested_error(error, context, step)
# Debugging and tracing
Pipeline.Debug.NestedExecution.start_debug_session(trace_context, options)
Pipeline.Tracing.NestedExecution.start_nested_trace(pipeline_id, context, step, parent)
Pipeline.Metrics.NestedPerformance.analyze_performance(execution_id)
```
### Configuration Options
```elixir
# config/config.exs
config :pipeline,
# Safety limits
max_nesting_depth: 10,
max_total_steps: 1000,
default_memory_limit_mb: 1024,
default_timeout_seconds: 300,
# Performance
pipeline_cache_ttl: 3600,
enable_tracing: false,
# Paths
nested_workspace_root: "./nested_workspaces",
allowed_pipeline_dirs: ["./pipelines", "./shared"],
# Features
enable_pipeline_registry: true,
enable_performance_metrics: true
```
## Related Guides
### Core Documentation
- **[Context Management Guide](docs/guides/context_management.md)** - Deep dive into variable passing and context inheritance
- **[Safety Features Guide](docs/guides/safety_features.md)** - Comprehensive safety configuration and best practices
- **Developer Experience Guide** - Debugging tools and performance optimization (see `docs/guides/developer_experience.md`)
- **Testing Strategy Guide** - Testing approaches for nested pipelines (see `docs/guides/testing_recursive_pipelines.md`)
- **Examples Collection** - Real-world use cases and patterns (see `docs/guides/recursive_pipeline_examples.md`)
### Technical References
- **Architecture Design** - Complete technical design document (see `docs/architecture/20250703_recursive.md`)
- **Implementation Prompts** - Phased implementation guide (see `docs/architecture/recursive_pipeline_prompts.md`)
- **Implementation Worklog** - Detailed implementation notes (see `docs/architecture/20250703_recursive_worklog.md`)
### Related Features
- **[Pipeline Organization](docs/architecture/pipeline_organization.md)** - Pipeline categorization and structure
---
## Implementation Status
### ✅ Phase 1: Core Infrastructure (Completed)
- Basic nested pipeline execution
- Three pipeline source options
- Context isolation and depth tracking
- Error propagation
- 14 tests with full coverage
### ✅ Phase 2: Context Management (Completed)
- Full template resolution engine
- Input mapping with type preservation
- Flexible output extraction
- Context inheritance options
- 21 additional tests
### ✅ Phase 3: Safety Features (Completed)
- Recursion and circular dependency protection
- Memory and timeout monitoring
- Resource cleanup guarantees
- Environment-specific configuration
- 46 safety-focused tests
### ✅ Phase 4: Developer Experience (Completed)
- Enhanced error messages with full context
- Span-based execution tracing
- Visual debugging tools
- Performance metrics and analysis
- 71 debugging and metrics tests
### 📋 Phase 5: Advanced Features (Future)
- Pipeline caching system
- Performance optimizations
- Pipeline registry
- Template system
### Current Capabilities
The recursive pipeline system is **production-ready** with:
- **Complete functionality** for nested pipeline execution
- **Enterprise-grade safety** protections
- **Comprehensive debugging** and monitoring tools
- **152+ tests** ensuring reliability
- **Full backward compatibility** with existing pipelines
---
This guide provides comprehensive documentation for the recursive pipelines feature. The system enables powerful pipeline composition patterns with production-grade safety, performance, and debugging capabilities.
