llama-chat VS llama-cpp-python

What are some alternatives?

promptfoo - Test your prompts. Evaluate and compare LLM outputs, catch regressions, and improve prompt quality. [Moved to: https://github.com/promptfoo/promptfoo]

LocalAI - :robot: The free, Open Source OpenAI alternative. Self-hosted, community-driven and local-first. Drop-in replacement for OpenAI running on consumer-grade hardware. No GPU required. Runs gguf, transformers, diffusers and many more models architectures. It allows to generate Text, Audio, Video, Images. Also with voice cloning capabilities.

TensorRT-LLM - TensorRT-LLM provides users with an easy-to-use Python API to define Large Language Models (LLMs) and build TensorRT engines that contain state-of-the-art optimizations to perform inference efficiently on NVIDIA GPUs. TensorRT-LLM also contains components to create Python and C++ runtimes that execute those TensorRT engines.

intel-extension-for-pytorch - A Python package for extending the official PyTorch that can easily obtain performance on Intel platform

mlx - MLX: An array framework for Apple silicon

llama.cpp - LLM inference in C/C++

cloudseeder - One-click install internet appliances that operate on your terms. Transform your home computer into a sovereign and secure cloud.

text-generation-inference - Large Language Model Text Generation Inference

ollama_local_rag

mlc-llm - Universal LLM Deployment Engine with ML Compilation

FastChat - An open platform for training, serving, and evaluating large language models. Release repo for Vicuna and Chatbot Arena.

ollama - Get up and running with Llama 3, Mistral, Gemma, and other large language models.