Python llm-serving

Open-source Python projects categorized as llm-serving
Edit details
Topics: llm Machine Learning Deep Learning Pytorch Python
SaaSHub - Software Alternatives and Reviews
SaaSHub helps you find the best software and product alternatives
www.saashub.com
featured

Top 5 Python llm-serving Projects

  • Ray

    Ray is a unified framework for scaling AI and Python applications. Ray consists of a core distributed runtime and a set of AI Libraries for accelerating ML workloads.

    • Project mention: Ray: Unified framework for scaling AI and Python applications | news.ycombinator.com | 2024-05-03

  • vllm

    A high-throughput and memory-efficient inference and serving engine for LLMs

    • Project mention: AI leaderboards are no longer useful. It's time to switch to Pareto curves | news.ycombinator.com | 2024-04-30

    I guess the root cause of my claim is that OpenAI won't tell us whether or not GPT-3.5 is an MoE model, and I assumed it wasn't. Since GPT-3.5 is clearly nondeterministic at temp=0, I believed the nondeterminism was due to FPU stuff, and this effect was amplified with GPT-4's MoE. But if GPT-3.5 is also MoE then that's just wrong.

    What makes this especially tricky is that small models are truly 100% deterministic at temp=0 because the relative likelihoods are too coarse for FPU issues to be a factor. I had thought 3.5 was big enough that some of its token probabilities were too fine-grained for the FPU. But that's probably wrong.

    On the other hand, it's not just GPT, there are currently floating-point difficulties in vllm which significantly affect the determinism of any model run on it: https://github.com/vllm-project/vllm/issues/966 Note that a suggested fix is upcasting to float32. So it's possible that GPT-3.5 is using an especially low-precision float and introducing nondeterminism by saving money on compute costs.

    Sadly I do not have the money[1] to actually run a test to falsify any of this. It seems like this would be a good little research project.

    [1] Or the time, or the motivation :) But this stuff is expensive.

  • InfluxDB

    Power Real-Time Data Analytics at Scale. Get real-time insights from all types of time series data with InfluxDB. Ingest, query, and analyze billions of data points in real-time with unbounded cardinality.

    InfluxDB logo

  • OpenLLM

    Run any open-source LLMs, such as Llama 2, Mistral, as OpenAI compatible API endpoint in the cloud.

    • Project mention: First 15 Open Source Advent projects | dev.to | 2023-12-15

    13. OpenLLM by BentoML | Github | tutorial

  • skypilot

    SkyPilot: Run LLMs, AI, and Batch jobs on any cloud. Get maximum savings, highest GPU availability, and managed execution—all with a simple interface.

    • Project mention: Alternative clouds are booming as companies seek cheaper access to GPUs | news.ycombinator.com | 2024-05-06

    Skypilot is worth a mention here:

    https://github.com/skypilot-org/skypilot

    Open source CLI to deploy multiple gpu vm’s on all major cloud providers, with an option to use spot pricing with 1 cheap vm used as a controller to always make sure you have the most inexpensive deployment available with failover and load balancing.

    It’s like beating the cloud providers at their own game I wouldn’t be surprised if they banned it.

  • mosec

    A high-performance ML model serving framework, offers dynamic batching and CPU/GPU pipelines to fully exploit your compute machine

    • Project mention: 20x Faster as the Beginning: Introducing pgvecto.rs extension written in Rust | dev.to | 2023-08-06

    Mosec - A high-performance serving framework for ML models, offers dynamic batching and CPU/GPU pipelines to fully exploit your compute machine. Simple and faster alternative to NVIDIA Triton.

  • SaaSHub

    SaaSHub - Software Alternatives and Reviews. SaaSHub helps you find the best software and product alternatives

    SaaSHub logo
NOTE: The open source projects on this list are ordered by number of github stars. The number of mentions indicates repo mentiontions in the last 12 Months or since we started tracking (Dec 2020).

Python llm-serving related posts

  • The Lost Arts of CLJS Frontend

    1 project | /r/Clojure | 13 Jul 2023

  • OpenLLM: An open platform for operating large language models (LLMs) in production.

    1 project | /r/u_sasizza | 7 Jul 2023

  • OpenLLM: An open platform for operating large language models (LLMs) in production.

    2 projects | /r/opensource | 6 Jul 2023

  • OpenLLM: An open platform for operating large language models (LLMs) in production.

    1 project | /r/softwarecrafters | 30 Jun 2023

  • GPT Weekly - 26the June Edition - 🎙️ Meta's Voicebox is Paused, 🖼️SDXL 0.9, 📜AI Compliance & EU Act and more

    2 projects | /r/ChatGPT | 26 Jun 2023

  • OpenLLM: OSS to easily serve Open Source LLMs

    1 project | /r/AutoGPT | 22 Jun 2023

  • OpenLLM: OSS to easily serve Open Source LLMs

    1 project | /r/aipromptprogramming | 21 Jun 2023
  • A note from our sponsor - SaaSHub
    www.saashub.com | 21 May 2024
    SaaSHub helps you find the best software and product alternatives Learn more →

Index

What are some of the best open-source llm-serving projects in Python? This list will help you:

Project Stars
1 Ray 31,414
2 vllm 19,672
3 OpenLLM 8,963
4 skypilot 5,729
5 mosec 712

Sponsored
SaaSHub - Software Alternatives and Reviews
SaaSHub helps you find the best software and product alternatives
www.saashub.com