Python Luigi Projects

luigi

14 17,407 6.3 Python

Luigi is a Python module that helps you build complex pipelines of batch jobs. It handles dependency resolution, workflow management, visualization etc. It also comes with Hadoop support built in.

Project mention: Ask HN: What is the correct way to deal with pipelines? | news.ycombinator.com | 2023-09-21

I agree there are many options in this space. Two others to consider:
- https://airflow.apache.org/
- https://github.com/spotify/luigi
There are also many Kubernetes based options out there. For the specific use case you specified, you might even consider a plain old Makefile and incrond if you expect these all to run on a single host and be triggered by a new file showing up in a directory…

Scout Monitoring

www.scoutapm.com featured

Free Django app performance insights with Scout Monitoring. Get Scout setup in minutes, and let us sweat the small stuff. A couple lines in settings.py is all you need to start monitoring your apps. Sign up for our free tier today.

NOTE: The open source projects on this list are ordered by number of github stars. The number of mentions indicates repo mentiontions in the last 12 Months or since we started tracking (Dec 2020).

Python Luigi related posts

In the context of Python what is a Bob Job?

2 projects | /r/learnpython | 10 Jul 2022
Apache Airflow. How to make the complex workflow as an easy job

1 project | dev.to | 20 Feb 2022
Open Source ETL Project For Startups

3 projects | dev.to | 22 Sep 2021
Resources/tutorials to help me learn about ETL?

1 project | /r/dataengineering | 29 Jun 2021
How to best integrate make-like functionality with invoke tasks/library?

1 project | /r/Python | 22 Mar 2021
Creating a scalable Flask Application

1 project | /r/flask | 26 Jan 2021
Spotify/luigi: build complex pipelines of batch jobs in Python

1 project | news.ycombinator.com | 15 Jan 2021
A note from our sponsor - SaaSHub
www.saashub.com | 3 Jun 2024

SaaSHub helps you find the best software and product alternatives Learn more →