Overview
Deppy is a lightweight Python library for building and executing dataflow pipelines using Directed Acyclic Graphs (DAGs). It allows developers to define dependencies between tasks and execute them efficiently. This library is published on pypi for public use.
Motivation
While Python offers powerful tools like async and multithreading. Deppy was created to provide a simple abstraction for constructing and executing data pipelines, making concurrent execution more accessible without requiring deep knowledge of async programming or multithreading.
Key Features
- DAG-based pipeline definition
- Automatic dependency resolution between tasks
- Efficient execution using async primitives
- Parallel execution where possible