Get real-time insights from all types of time series data with InfluxDB. Ingest, query, and analyze billions of data points in real-time with unbounded cardinality. Learn more →
Top 4 Python inference-server Projects
-
inference
A fast, easy-to-use, production-ready inference server for computer vision supporting deployment of many popular model architectures and fine-tuned models. (by roboflow)
-
-
Scout Monitoring
Free Django app performance insights with Scout Monitoring. Get Scout setup in minutes, and let us sweat the small stuff. A couple lines in settings.py is all you need to start monitoring your apps. Sign up for our free tier today.
-
pinferencia
Python + Inference - Model Deployment library in Python. Simplest model inference server ever.
-
inference-benchmark
Benchmark for machine learning model online serving (LLM, embedding, Stable-Diffusion, Whisper)
Great question! I work for a computer vision company (Roboflow) and have seen computer vision used for everything from accident prevention on critical infrastructure to identifying defects on vehicle parts to detecting trading cards for use in video game applications.
Drawing bounding boxes is a common end point for demos, but for businesses using computer vision there is an entire world after that: on device deployment. This can be on devices like an NVIDIA Jetson (a very common choice), to Raspberry Pis to central CUDA GPU servers for processing large volumes of data (maybe connected to cameras over RTSP).
Note: There are many models that are faster and perform better than YOLOv5 (i.e. YOLOv8, YOLOv10, PaliGemma). Roboflow Inference that our ML team maintains has various guides on deploying models to the edge: https://inference.roboflow.com/#inference-pipeline
I have done some benchmarks before: https://github.com/tensorchord/inference-benchmark
NOTE:
The open source projects on this list are ordered by number of github stars.
The number of mentions indicates repo mentiontions in the last 12 Months or
since we started tracking (Dec 2020).
Python inference-server related posts
-
Show HN: Pinferencia, Deploy Your AI Models with Pretty UI and REST API
-
Stop Writing Flask to Serve/Deploy Your Model: Pinferencia is Here
-
Looking for a reference design pattern for an image to image microservice
-
Google T5 Translation as a Service with Just 7 lines of Codes
-
Pre-trained Model with Fine Tuning/Transfer Learning or Design and Train from Scratch?
-
[D] Pre-trained Model with Fine Tuning/Transfer Learning or Design and Train from Scratch?
-
GPT2 — Text Generation Transformer: How to Use & How to Serve
-
A note from our sponsor - InfluxDB
www.influxdata.com | 2 Jun 2024
Index
What are some of the best open-source inference-server projects in Python? This list will help you:
| Project | Stars | |
|---|---|---|
| 1 | inference | 1,079 |
| 2 | truss | 848 |
| 3 | pinferencia | 558 |
| 4 | inference-benchmark | 26 |
Sponsored
SaaSHub - Software Alternatives and Reviews
SaaSHub helps you find the best software and product alternatives
www.saashub.com