wpipe

📚 Documentation

  • Getting Started with wpipe
    • 1. Installation
    • 2. Verification
    • 3. Build Your First Pipeline
    • 4. The “Hello World” of Orchestration
    • 5. Enterprise Capabilities at a Glance
    • Next Steps
  • Installation Guide
    • 1. Requirements
    • 2. Standard Installation
    • 3. Advanced Installation
    • 4. Containerized (Docker)
    • 5. Verification
    • 6. Troubleshooting
  • Mastering wpipe: Usage Guide
    • 1. Fundamental Step Types
    • 2. Advanced Control Flow
    • 3. Enterprise Resiliency
    • 4. High Performance
    • 5. Operational Integration
  • User Guide
    • 1. Getting Started
      • Pipeline Basics
    • 2. Core Features
      • Conditions
      • Retry Logic
      • API Integration
      • SQLite Storage
      • YAML Configuration
      • Nested Pipelines
      • Error Handling
    • 3. Advanced Topics
      • Best Practices
      • Troubleshooting
        • Pipeline Basics
        • Conditional Branching
        • Retry Logic
        • API Integration
        • SQLite Storage
        • YAML Configuration
        • Nested Pipelines
        • Error Handling
        • Best Practices
        • Troubleshooting
  • WPipe Academy: Master Your Pipelines
    • La Senda del Maestro: 140 Niveles de Poder
      • Listado Detallado de Niveles
        • Nivel 1: Conceptos Básicos de Pipeline
        • Nivel 2: Metadatos y Decorador @step
        • Nivel 3: demo_level3.py
        • Nivel 4: demo_level4.py
        • Nivel 5: demo_level5.py
        • Nivel 6: demo_level6.py
        • Nivel 7: demo_level7.py
        • Nivel 8: demo_level8.py
        • Nivel 9: demo_level9.py
        • Nivel 10: demo_level10.py
        • Nivel 11: demo_level11.py
        • Nivel 12: demo_level12.py
        • Nivel 13: demo_level13.py
        • Nivel 14: demo_level14.py
        • Nivel 15: demo_level15.py
        • Nivel 16: demo_level16.py
        • Nivel 17: demo_level17.py
        • Nivel 18: demo_level18.py
        • Nivel 19: demo_level19.py
        • Nivel 20: demo_level20.py
        • Nivel 21: demo_level21.py
        • Nivel 22: demo_level22.py
        • Nivel 23: demo_level23.py
        • Nivel 24: demo_level24.py
        • Nivel 25: Super-Computación (Threads vs Processes)
        • Nivel 26: demo_level26.py
        • Nivel 27: demo_level27.py
        • Nivel 28: demo_level28.py
        • Nivel 29: demo_level29.py
        • Nivel 30: demo_level30.py
        • Nivel 31: demo_level31.py
        • Nivel 32: demo_level32.py
        • Nivel 33: demo_level33.py
        • Nivel 34: demo_level34.py
        • Nivel 35: demo_level35.py
        • Nivel 36: demo_level36.py
        • Nivel 37: demo_level37.py
        • Nivel 38: demo_level38.py
        • Nivel 39: demo_level39.py
        • Nivel 40: demo_level40.py
        • Nivel 41: demo_level41.py
        • Nivel 42: demo_level42.py
        • Nivel 43: demo_level43.py
        • Nivel 44: demo_level44.py
        • Nivel 45: demo_level45.py
        • Nivel 46: demo_level46.py
        • Nivel 47: demo_level47.py
        • Nivel 48: demo_level48.py
        • Nivel 49: demo_level49.py
        • Nivel 50: Integración Final - El Viaje Completo
        • Nivel 51: demo_level51.py
        • Nivel 52: demo_level52.py
        • Nivel 53: demo_level53.py
        • Nivel 54: demo_level54.py
        • Nivel 55: demo_level55.py
        • Nivel 56: demo_level56.py
        • Nivel 57: demo_level57.py
        • Nivel 58: demo_level58.py
        • Nivel 59: demo_level59.py
        • Nivel 60: demo_level60.py
        • Nivel 61: demo_level61.py
        • Nivel 62: demo_level62.py
        • Nivel 63: demo_level63.py
        • Nivel 64: demo_level64.py
        • Nivel 65: demo_level65.py
        • Nivel 66: demo_level66.py
        • Nivel 67: demo_level67.py
        • Nivel 68: demo_level68.py
        • Nivel 69: demo_level69.py
        • Nivel 70: demo_level70.py
        • Nivel 71: demo_level71.py
        • Nivel 72: demo_level72.py
        • Nivel 73: demo_level73.py
        • Nivel 74: demo_level74.py
        • Nivel 75: Orquestación Híbrida (For + Parallel)
        • Nivel 76: demo_level76.py
        • Nivel 77: demo_level77.py
        • Nivel 78: demo_level78.py
        • Nivel 79: demo_level79.py
        • Nivel 80: demo_level80.py
        • Nivel 81: demo_level81.py
        • Nivel 82: demo_level82.py
        • Nivel 83: demo_level83.py
        • Nivel 84: demo_level84.py
        • Nivel 85: demo_level85.py
        • Nivel 86: demo_level86.py
        • Nivel 87: demo_level87.py
        • Nivel 88: demo_level88.py
        • Nivel 89: demo_level89.py
        • Nivel 90: demo_level90.py
        • Nivel 91: demo_level91.py
        • Nivel 92: demo_level92.py
        • Nivel 93: demo_level93.py
        • Nivel 94: demo_level94.py
        • Nivel 95: demo_level95.py
        • Nivel 96: demo_level96.py
        • Nivel 97: demo_level97.py
        • Nivel 98: demo_level98.py
        • Nivel 99: demo_level99.py
        • Nivel 100: Auditoría de Seguridad (Alerts History)
        • Nivel 101: demo_level101.py
        • Nivel 102: demo_level102.py
        • Nivel 103: demo_level103.py
        • Nivel 104: demo_level104.py
        • Nivel 105: demo_level105.py
        • Nivel 106: demo_level106.py
        • Nivel 107: demo_level107.py
        • Nivel 108: demo_level108.py
        • Nivel 109: demo_level109.py
        • Nivel 110: demo_level110.py
        • Nivel 111: demo_level111.py
        • Nivel 112: demo_level112.py
        • Nivel 113: demo_level113.py
        • Nivel 114: demo_level114.py
        • Nivel 115: demo_level115.py
        • Nivel 116: demo_level116.py
        • Nivel 117: demo_level117.py
        • Nivel 118: demo_level118.py
        • Nivel 119: demo_level119.py
        • Nivel 120: demo_level120.py
        • Nivel 121: demo_level121.py
        • Nivel 122: demo_level122.py
        • Nivel 123: demo_level123.py
        • Nivel 124: demo_level124.py
        • Nivel 125: demo_level125.py
        • Nivel 126: demo_level126.py
        • Nivel 127: demo_level127.py
        • Nivel 128: demo_level128.py
        • Nivel 129: demo_level129.py
        • Nivel 130: Integración de API y Métricas de Negocio
        • Nivel 131: Background Tasks - Ejecución No Bloqueante
        • Nivel 132: Background con Captura de Errores
        • Nivel 133: Múltiples Background Tasks
        • Nivel 134: Background con Pipeline Anidado
        • Nivel 135: Background para Logging/Telemetría
        • Nivel 136: Background en PipelineAsync
        • Nivel 137: Background con Tuple Steps
        • Nivel 138: Background dentro de Condition
        • Nivel 139: Background sin capture_error (Default)
        • Nivel 140: El Zen de WPipe - Demo Final de Background Tasks
        • 📜 Próximos Pasos
    • Basic Pipeline Tutorial
      • 1. Introduction and Overview
        • 1.1 What is a Pipeline?
        • 1.2 What You’ll Build
      • 2. Environment Setup
        • 2.1 Prerequisites
        • 2.2 Installation
      • 3. Your First Pipeline
        • 3.1 Basic Structure
        • 3.2 Step-by-Step Implementation
      • 4. Understanding Data Flow
        • 4.1 How Data Accumulates
        • 4.2 Practical Example of Data Evolution
      • 5. Passing Initial Data
        • 5.1 Providing Input Data
        • 5.2 Use Case: Parameterized Pipelines
      • 6. Step Function Patterns
        • 6.1 Lambda Functions
        • 6.2 Class-Based Steps
      • 7. Error Handling
        • 7.1 Basic Error Handling
        • 7.2 Validation Errors
      • 8. Advanced Features
        • 8.1 Conditional Execution
        • 8.2 Retry Logic
      • 9. Testing Your Pipeline
        • 9.1 Unit Testing Steps
        • 9.2 Integration Testing
      • 10. Best Practices Summary
      • 11. Exercises
        • 11.1 Exercise 1: String Processing Pipeline
        • 11.2 Exercise 2: Temperature Converter
        • 11.3 Exercise 3: Shopping Cart
      • 12. Next Steps
    • Class-Based Steps Tutorial
      • 1. Introduction
      • 2. The __call__ Protocol
      • 3. Creating Step Classes
        • 3.1 Basic Step Class
        • 3.2 Configurable Step Class
      • 4. Stateful Steps
        • 4.1 Running Total
        • 4.2 Accumulator Pattern
      • 5. Composable Steps
        • 5.1 Pipeline Building Blocks
      • 6. Class Methods as Steps
      • 7. Best Practices
        • 7.1 Keep Steps Focused
        • 7.2 Use Type Hints
        • 7.3 Document Your Classes
      • 8. Advanced Patterns
        • 8.1 Step Factory
        • 8.2 Decorator Pattern
      • 9. Complete Example
      • 10. Next Steps
    • API Integration Tutorial
      • 1. Introduction
      • 2. Understanding the API Client
      • 3. Worker Registration
        • 3.1 Basic Registration
        • 3.2 Setting Worker ID
      • 4. Process Tracking
        • 4.1 Starting a Process
        • 4.2 Process Flow
      • 5. API Methods Reference
        • 5.1 Worker Methods
        • 5.2 Process Methods
        • 5.3 Task Methods
      • 6. Custom API Implementation
        • 6.1 Extending APIClient
      • 7. Error Handling with API
        • 7.1 Graceful Degradation
      • 8. Complete Example
      • 9. Testing API Integration
        • 9.1 Mock Server
      • 10. Security Best Practices
        • 10.1 Token Management
        • 10.2 HTTPS Only
      • 11. Troubleshooting
        • 11.1 Common Issues
        • 11.2 Debug Mode
      • 12. Next Steps
    • Error Handling Tutorial
      • 1. Introduction
      • 2. Understanding Exceptions
        • 2.1 TaskError
        • 2.2 Error Codes
      • 3. Raising TaskError
        • 3.1 Basic Usage
        • 3.2 With Step Information
      • 4. Catching Errors
        • 4.1 Basic Catch
        • 4.2 Specific Error Codes
      • 5. Error Recovery
        • 5.1 Graceful Degradation
        • 5.2 Fallback Values
      • 6. Validation Patterns
        • 6.1 Early Validation
        • 6.2 Chain Validation
      • 7. API Error Handling
        • 7.1 Handle API Failures
        • 7.2 Timeout Handling
      • 8. Custom Error Classes
        • 8.1 Define Custom Errors
        • 8.2 Use Custom Errors
      • 9. Error Logging
        • 9.1 Log Errors
        • 9.2 Error Context
      • 10. Complete Example
      • 11. Best Practices
      • 12. Next Steps
    • Retry Logic Tutorial
      • 1. Introduction
      • 2. Built-in Retry Support
        • 2.1 Configuring Retries
        • 2.2 How It Works
      • 3. Custom Retry Decorators
        • 3.1 Basic Retry Decorator
        • 3.2 Using the Decorator
      • 4. Advanced Retry Patterns
        • 4.1 Retry with Custom Logic
        • 4.2 Retry with Logging
      • 5. Step-Level Retries
        • 5.1 Independent Retry Config
      • 6. Handling Different Failure Types
        • 6.1 Network Failures
        • 6.2 Database Failures
      • 7. Timeout Handling
        • 7.1 Combined Retry and Timeout
      • 8. Complete Example
      • 9. Best Practices
      • 10. Next Steps
    • Nested Pipelines Tutorial
      • 1. Introduction
      • 2. Basic Nested Pipeline
        • 2.1 Creating Nested Pipelines
      • 3. Pipeline Factory Pattern
        • 3.1 Creating Reusable Pipelines
      • 4. Complete Example
      • 5. Best Practices
      • 6. Next Steps
    • SQLite Integration Tutorial
      • 1. Introduction
      • 2. Basic SQLite Usage
        • 2.1 Using Wsqlite
      • 3. Querying Results
        • 3.1 Reading Records
      • 4. Advanced Patterns
        • 4.1 Multiple Executions
      • 5. Complete Example
      • 6. Best Practices
      • 7. Next Steps
    • YAML Configuration Tutorial
      • 1. Introduction
      • 2. Basic YAML Usage
        • 2.1 Creating Configuration File
        • 2.2 Loading Configuration
      • 3. Environment Variables
        • 3.1 Using env vars in YAML
      • 4. Complete Example
      • 5. Best Practices
      • 6. Next Steps
    • Conditional Branches Tutorial
      • 1. Introduction
        • 1.1 When to Use Conditions
        • 1.2 Basic Concept
      • 2. Condition Class Reference
      • 3. Basic Examples
        • 3.1 Simple Boolean Check
        • 3.2 String Matching
        • 3.3 Without Else Branch
      • 4. Complex Conditions
        • 4.1 Multiple Comparisons
        • 4.2 Numeric Ranges
        • 4.3 In Operator
        • 4.4 String Contains
      • 5. Nested Conditions
        • 5.1 Chained Conditions
        • 5.2 Multiple Independent Conditions
      • 6. Real-World Examples
        • 6.1 User Registration Flow
        • 6.2 Order Processing
        • 6.3 Data Validation Pipeline
      • 7. Best Practices
        • 7.1 Keep Expressions Simple
        • 7.2 Use Meaningful Variable Names
        • 7.3 Order Conditions Strategically
        • 7.4 Test Both Branches
      • 8. Troubleshooting
        • 8.1 Condition Not Evaluating
        • 8.2 Type Mismatch
      • 9. Complete Example
      • 10. Next Steps
    • Advanced Patterns Tutorial
      • 1. DAG Scheduling (Directed Acyclic Graphs)
      • 2. Native Parallelism (IO vs CPU)
      • 3. Forensic Error Capture
      • 4. High-Performance Monitoring
      • 5. Dynamic Pipeline Composition
    • Production Deployment Tutorial
      • 1. Environment Setup
        • 1.1 Production Requirements
      • 2. Configuration Management
        • 2.1 Environment-Based Config
      • 3. Logging and Monitoring
        • 3.1 Structured Logging
      • 4. Error Handling in Production
        • 4.1 Graceful Degradation
      • 5. Health Checks
        • 5.1 Health Check Endpoint
      • 6. CI/CD Integration
        • 6.1 GitHub Actions Example
      • 7. Docker Deployment
        • 7.1 Dockerfile Example
      • 8. Complete Example
      • 9. Best Practices
      • 10. Next Steps
    • The Learning Tour
    • Core Curriculum
    • Recommended Learning Plan
    • Troubleshooting During Tutorials
    • Next Steps
  • API Technical Specification
    • 1. Core Orchestration
    • 2. Logical Control Blocks
    • 3. Decorators & Intelligence
    • 4. Persistence (WSQLite)
    • 5. Industrial Observability
    • 6. Error Architecture
    • 7. Technical Utilities
  • Examples Gallery
    • 01 Basic Pipeline (15 Examples)
    • 02 API Pipeline (20 Examples)
    • 03 Error Handling (15 Examples)
    • 04 Conditional Branching (12 Examples)
    • 05 Retry Logic (12 Examples)
    • 06 SQLite Integration (14 Examples)
    • 07 Nested Pipelines (14 Examples)
    • 08 YAML Configuration (14 Examples)
    • 09 Microservice (11 Examples)
    • Running Examples
    • Example Directory Structure
  • System Architecture: The WPipe Engine
    • 1. The “Warehouse” Model
    • 2. Core Internal Layers
    • 3. Persistence Strategy: WSQLite
    • 4. Resiliency & Reliability
      • 4.1 Smart Checkpointing
      • 4.2 Forensic Error Capture
    • 5. Concurrency Model
    • 6. Module Hierarchy
    • Next Steps
  • Best Practices for Industrial Pipelines
    • 1. Step Architecture: Atomic & Pure
    • 2. Data Contract Management
    • 3. Resiliency & Failure Strategy
    • 4. Performance & Scalability
    • 5. Monitoring & Governance
  • FAQ
    • 1. General Questions
      • 1.1 What is wpipe?
      • 1.2 What are the main features?
      • 1.3 What is the license?
      • 1.4 Is wpipe production-ready?
    • 2. Installation Questions
      • 2.1 What Python version is required?
      • 2.2 How do I install wpipe?
      • 2.3 How do I install from source?
      • 2.4 What are the dependencies?
    • 3. Getting Started Questions
      • 3.1 How do I create a basic pipeline?
      • 3.2 What is the data flow between steps?
      • 3.3 Can I use classes as steps?
      • 3.4 How do I run a pipeline?
    • 4. API Integration Questions
      • 4.1 How do I connect to an API?
      • 4.2 How do I register a worker?
      • 4.3 How do I perform health checks?
    • 5. SQLite Questions
      • 5.1 How do I persist results?
      • 5.2 What data is stored?
      • 5.3 How do I query results?
    • 6. Error Handling Questions
      • 6.1 How do I handle errors?
      • 6.2 What error codes are available?
      • 6.3 How do I create custom error codes?
    • 7. Configuration Questions
      • 7.1 How do I use YAML configuration?
      • 7.2 Can I use environment variables?
    • 8. Troubleshooting Questions
      • 8.1 My pipeline is not connecting to the API
      • 8.2 The pipeline is stuck on a step
      • 8.3 I’m getting TaskError exceptions
      • 8.4 How do I enable verbose output?
    • 9. Performance Questions
      • 9.1 How fast is wpipe?
      • 9.2 Can I process large datasets?
      • 9.3 How do I monitor memory usage?
    • 10. Advanced Questions
      • 10.1 Can I create conditional branches?
      • 10.2 Can I nest pipelines?
      • 10.3 Can I add callbacks?
    • 11. Contributing Questions
      • 11.1 How do I report bugs?
      • 11.2 How do I contribute?
      • 11.3 Where can I get help?
  • Glossary
    • 1. Core Concepts
      • 1.1 Pipeline
      • 1.2 Step
      • 1.3 Data Flow
      • 1.4 Step Name
      • 1.5 Step Version
    • 2. API Integration
      • 2.1 API Client
      • 2.2 Worker
      • 2.3 Worker ID
      • 2.4 Health Check
      • 2.5 Task Status
    • 3. Data Persistence
      • 3.1 SQLite
      • 3.2 Execution Record
      • 3.3 Wsqlite
    • 4. Error Handling
      • 4.1 TaskError
      • 4.2 Error Codes
      • 4.3 Validation Error
      • 4.4 Retryable Error
    • 5. Configuration
      • 5.1 YAML Configuration
      • 5.2 Environment Variable
      • 5.3 Default Value
    • 6. Progress and Logging
      • 6.1 Verbose Mode
      • 6.2 Progress Manager
      • 6.3 Logger
    • 7. Advanced Concepts
      • 7.1 Nested Pipeline
      • 7.2 Condition
      • 7.3 Decorator
      • 7.4 Callback
      • 7.5 Idempotent Step
    • 8. Testing Terms
      • 8.1 Unit Test
      • 8.2 Integration Test
      • 8.3 Mock
    • 9. Acronyms
    • 10. Related Libraries
  • Contributing
    • 1. Getting Started
      • 1.1 Fork the Repository
      • 1.2 Set Up Development Environment
      • 1.3 Run Tests
    • 2. Development Workflow
      • 2.1 Create a Branch
      • 2.2 Make Your Changes
      • 2.3 Run Quality Checks
    • 3. Testing Guidelines
      • 3.1 Write Tests
      • 3.2 Test File Naming
      • 3.3 Run Specific Tests
    • 4. Documentation
      • 4.1 Update Documentation
      • 4.2 Build Documentation
      • 4.3 Documentation Style
    • 5. Commit Guidelines
      • 5.1 Commit Messages
      • 5.2 Commit Message Format
    • 6. Pull Request Process
      • 6.1 Create Pull Request
      • 6.2 Pull Request Template
      • 6.3 What Happens Next
    • 7. Reporting Issues
      • 7.1 Bug Reports
      • 7.2 Feature Requests
    • 8. Code of Conduct
    • 9. Questions?
  • Changelog
    • [1.5.3] - 2026-04-13
      • Added
    • [1.0.0] - 2026-03-22
      • Added
    • [0.1.7] - 2024-12-15
      • Added
      • Fixed
    • [0.1.6] - 2024-11-20
      • Added
    • [0.1.5] - 2024-10-10
      • Added
    • [0.1.0] - 2024-01-01
      • Added
wpipe
  • Search

© Copyright 2024-2026, William Steve Rodriguez Villamizar.

Built with Sphinx using a theme provided by Read the Docs.