How do you deal with parallelising parts of an ML pipeline especially on Python?

• ploomber

  121 3,396 7.4 Python

  The fastest ⚡️ way to build data pipelines. Develop iteratively, deploy anywhere. ☁️

  

Multiprocessing works well but you probably need an abstraction on top to make it work reliably. For starters, it's best to use a pool of processes because creating new ones is expensive, you also need to ensure that errors in the sub-processes are correctly displayed in the main process, otherwise, it becomes frustrating. Also, sometimes sub-processings might get stuck so you have to monitor them. I implemented something that takes care of all of that for a project I'm working on, it'll give you an idea of what it looks like (of course, you can use the framework as well, which lets you parallelize functions and notebooks).

• debuglater

  8 56 3.8 Python

  Store Python traceback for later debugging. 🐛

  

Finally, debugging. If you're running code in sub-processes; debugging becomes a real pain because out of the box, you won't be able to start a debugger in the sub-processes. Furthermore, there's a chance that more than one fails. One solution is to dump the traceback when any sub-process fails, so you can start a debugging sesstion afterward; look at this project for an example.

• Scout Monitoring

  www.scoutapm.com featured

  Free Django app performance insights with Scout Monitoring. Get Scout setup in minutes, and let us sweat the small stuff. A couple lines in settings.py is all you need to start monitoring your apps. Sign up for our free tier today.

  

• mpire

  8 1,930 7.3 Python

  A Python package for easy multiprocessing, but faster than multiprocessing

  

https://github.com/Slimmer-AI/mpire is a nice lib, with better performance than multiprocessing.

• orchest

  44 4,023 4.5 TypeScript

  Build data pipelines, the easy way 🛠️

  

We automatically provide container level parallelism in Orchest: https://github.com/orchest/orchest

Suggest a related project