Examples Gallery
Explore 100+ examples organized by functionality. Each example includes code, explanation, and is located in the examples/ directory of the repository.
01 Basic Pipeline (15 Examples)
Simple pipelines with functions, classes, and data flow.
Location: examples/01_basic_pipeline/
These examples demonstrate the core concepts of wpipe pipelines.
Example |
Description |
|---|---|
Basic pipeline with a single function step |
|
Using class instances with |
|
Combining functions and classes in the same pipeline |
|
Handling missing data with default values |
|
Passing additional arguments to steps |
|
Advanced dictionary manipulation in steps |
|
Running the same pipeline with different data |
|
Aggregating data from multiple sources |
|
Handling empty input data gracefully |
|
Using lambda functions as simple steps |
|
Using decorators to modify step behavior |
|
Using context managers with pipelines |
|
Async pipeline execution patterns |
|
Chaining multiple pipelines together |
|
Cloning and modifying pipelines |
Quick Example:
from wpipe import Pipeline
def fetch_data(data):
return {"data": [1, 2, 3]}
def process(data):
return {"sum": sum(data["data"])}
pipeline = Pipeline(verbose=True)
pipeline.set_steps([
(fetch_data, "Fetch Data", "v1.0"),
(process, "Process Data", "v1.0"),
])
result = pipeline.run({})
02 API Pipeline (20 Examples)
Connect pipelines to external APIs for tracking and monitoring.
Location: examples/02_api_pipeline/
These examples show how to integrate with external APIs and track pipeline execution.
Example |
Description |
|---|---|
Basic API connection and configuration |
|
Setting and managing worker IDs |
|
Running without API (local only) |
|
Handling API connection errors gracefully |
|
Displaying detailed error information |
|
Configuring request timeouts |
|
Complete API configuration example |
|
Configuring retries for API calls |
|
Adding custom headers to requests |
|
Logging API interactions |
|
Attaching metadata to workers |
|
Handling API rate limits |
|
Processing data in batches via API |
|
Different authentication methods |
|
Implementing health check endpoints |
|
Dynamic service discovery |
|
Handling expired authentication tokens |
|
Network timeout handling |
|
Handling invalid API URLs |
|
Multiple workers running concurrently |
|
Automatic reconnection patterns |
Quick Example:
from wpipe import Pipeline
api_config = {
"base_url": "https://api.example.com",
"token": "your-auth-token"
}
pipeline = Pipeline(api_config=api_config)
pipeline.worker_register(name="processor", version="1.0.0")
03 Error Handling (15 Examples)
Robust error handling patterns and recovery strategies.
Location: examples/03_error_handling/
Learn how to handle errors gracefully and implement recovery patterns.
Example |
Description |
|---|---|
Catching and handling basic errors |
|
Different exception types in wpipe |
|
Working with TaskError exceptions |
|
Errors occurring mid-pipeline |
|
Continuing after non-critical errors |
|
Exception chaining and context |
|
Creating custom error types |
|
Recovery from errors |
|
Accessing partial results on failure |
|
Working with error codes |
|
Error context and debugging |
|
Complete error handling example |
|
Custom error definitions |
Quick Example:
from wpipe import Pipeline
from wpipe.exception import TaskError, Codes
def risky_step(data):
raise ConnectionError("Network unavailable")
pipeline = Pipeline(verbose=True)
try:
result = pipeline.run({})
except TaskError as e:
print(f"Error code: {e.error_code}")
print(f"Message: {str(e)}")
04 Conditional Branching (12 Examples)
Execute different paths based on data conditions.
Location: examples/04_condition/
Build complex decision trees with conditional execution.
Example |
Description |
|---|---|
Basic conditional branching |
|
String-based conditions |
|
Multiple steps in conditional branches |
|
Condition without else branch |
|
Handling invalid expressions |
|
Complex boolean expressions |
|
Numeric comparisons |
|
Equality checks in conditions |
|
Chaining multiple conditions |
|
Boolean logic in conditions |
|
Checking for None values |
Quick Example:
from wpipe import Pipeline, Condition
def check_value(data):
return {"value": 75}
def process_high(data):
return {"result": "High value"}
def process_low(data):
return {"result": "Low value"}
condition = Condition(
expression="value > 50",
branch_true=[(process_high, "Process High", "v1.0")],
branch_false=[(process_low, "Process Low", "v1.0")],
)
pipeline = Pipeline(verbose=True)
pipeline.set_steps([
(check_value, "Check Value", "v1.0"),
condition,
])
result = pipeline.run({})
05 Retry Logic (12 Examples)
Automatic retries for failed operations with backoff strategies.
Location: examples/05_retry/
Implement robust retry mechanisms for unreliable operations.
Example |
Description |
|---|---|
Basic retry configuration |
|
Succeeding after retries |
|
Filtering which exceptions trigger retries |
|
Retry across multiple steps |
|
Disabling retries |
|
Exponential backoff between retries |
|
Custom exception handling in retries |
|
Handling partial failures |
|
Tracking retry attempts |
|
Retry context management |
|
Managing state during retries |
Quick Example:
from wpipe import Pipeline
pipeline = Pipeline(
verbose=True,
max_retries=3,
retry_delay=2.0,
retry_on_exceptions=(ConnectionError, TimeoutError),
)
06 SQLite Integration (14 Examples)
Persist pipeline results to SQLite databases.
Location: examples/06_sqlite_integration/
Store and query pipeline execution results.
Example |
Description |
|---|---|
Writing results to database |
|
Using Wsqlite context manager |
|
Exporting data to CSV |
|
Advanced query patterns |
|
Batch insert operations |
|
Querying specific records |
|
Updating existing records |
|
Deleting records |
|
Complex query examples |
|
Storing JSON data |
|
Transaction handling |
|
Database utility functions |
Quick Example:
from wpipe import Pipeline
from wpipe.sqlite import Wsqlite
with Wsqlite(db_name="results.db") as db:
db.input = {"x": 10}
result = pipeline.run({"x": 10})
db.output = result
print(f"Record ID: {db.id}")
07 Nested Pipelines (14 Examples)
Compose complex workflows from smaller, reusable pipelines.
Location: examples/07_nested_pipelines/
Build modular, maintainable pipeline systems.
Example |
Description |
|---|---|
Basic nested pipeline structure |
|
Multiple nested pipelines |
|
Passing data between nested pipelines |
|
Reusing pipeline definitions |
|
Nested pipelines with data |
|
Deeply nested pipeline hierarchies |
|
Parallel nested execution |
|
Conditionals in nested pipelines |
|
Managing state in nested pipelines |
|
Error handling in nested pipelines |
|
Recursive pipeline patterns |
|
Nested pipeline utilities |
Quick Example:
from wpipe import Pipeline
sub_pipeline = Pipeline()
sub_pipeline.set_steps([
(step_a, "Step A", "v1.0"),
(step_b, "Step B", "v1.0"),
])
main_pipeline = Pipeline()
main_pipeline.set_steps([
(sub_pipeline.run, "Run Sub-pipeline", "v1.0"),
(step_c, "Step C", "v1.0"),
])
08 YAML Configuration (14 Examples)
Load and manage pipeline configuration from YAML files.
Location: examples/08_yaml_config/
Dynamic configuration and environment management.
Example |
Description |
|---|---|
Reading YAML configuration |
|
Writing YAML configuration |
|
Pipeline with YAML config |
|
Complex configuration structures |
|
Nested configuration options |
|
Loading steps from config |
|
Dynamic configuration loading |
|
Validating configuration |
|
Environment variable integration |
|
Multi-file configuration |
|
Nested env config |
|
Configuration loader utility |
Quick Example:
from wpipe import Pipeline
from wpipe.util import leer_yaml
config = leer_yaml("config.yaml")
pipeline = Pipeline(**config["pipeline"])
09 Microservice (11 Examples)
Build microservice workflows with wpipe.
Location: examples/09_microservice/
Production-ready microservice patterns.
Example |
Description |
|---|---|
Basic microservice setup |
|
Message processing patterns |
|
Service with integrated pipeline |
|
Health check implementation |
|
Service state management |
|
Input validation in services |
|
Metrics collection |
|
Service configuration |
|
Managing service dependencies |
|
Graceful shutdown patterns |
Quick Example:
from wpipe import Pipeline
pipeline = Pipeline(verbose=True)
pipeline.worker_register(name="microservice", version="1.0.0")
pipeline.set_steps([
(process_request, "Process Request", "v1.0"),
(validate_response, "Validate Response", "v1.0"),
])
Running Examples
Clone and run examples locally:
git clone https://github.com/wisrovi/wpipe.git
cd wpipe/examples/01_basic_pipeline/01_simple_function
python example.py
Each example folder contains:
example.py- Runnable example codeREADME.md- Detailed explanation with examples
Example Directory Structure
examples/
├── 01_basic_pipeline/ # 15 examples - Core pipeline functionality
├── 02_api_pipeline/ # 20 examples - API integration
├── 03_error_handling/ # 15 examples - Error handling
├── 04_condition/ # 12 examples - Conditional branching
├── 05_retry/ # 12 examples - Retry mechanisms
├── 06_sqlite_integration/ # 14 examples - Database operations
├── 07_nested_pipelines/ # 14 examples - Nested workflows
├── 08_yaml_config/ # 14 examples - Configuration
├── 09_microservice/ # 11 examples - Microservice patterns
└── test/ # Integration tests
Total: 100+ examples