llm_utils VS litellm

> tiktoken.encoding_for_model(model)

Calling this where model == 'gpt-4o' will encode with CL200k no?

But yes, I do agree with you. I had time implementing non-tiktoken tokenizers for my project. I ended up manually adding tokenizer.json files into my repo.[1] The other options is downloading from HF, but the official repos where the model's tokenizer.json lives require agreeing to their terms to access. So it requires an HF key, and agreeing to the terms. So not a good experience for a consumer of the package.

> Message frame tokens?

Do you mean the chat template tokens? Oh, that's another good point. Yeah, it counts OpenAI prompt tokens. I solved this by implementing a Jinja templating engine to create the full prompt. [2] Granted, both llama.cpp and mistral-rs do this on the backend, so it's purely for counting tokens. I guess it would make sense to add a function to convert tokens to Dollars.

[1] https://github.com/ShelbyJenkins/llm_utils/tree/main/src/mod...

Very cool! Is this cost directory you're using the best source for historical cost per 1M tokens? https://github.com/BerriAI/litellm/blob/main/model_prices_an...

Next, we'll show how an Embeddings database can integrate with external API services via LiteLLM .

There are a few libs that already abstract this away, for example:

- https://github.com/BerriAI/litellm

- https://jxnl.github.io/instructor/

- langchain

It's not hard for me to imagine a future where there is something like the CNCF for AI models, tools, and infra.

Hey, are you just looking for litellm - https://github.com/BerriAI/litellm

context - i'm the repo maintainer

It should be possible using LiteLLM and a patch or a proxy.

https://github.com/BerriAI/litellm

This article will demonstrate how txtai can integrate with llama.cpp, LiteLLM and custom generation methods. For custom generation, we'll show how to run inference with a Mamba model.

I use this with ollama and works perfectly https://github.com/BerriAI/litellm

Another abstraction layer library is: https://github.com/BerriAI/litellm

For me the killer feature of a library like this would be if it implemented function calling. Even if it was for a very restricted grammar - like the traditional ReAct prompt:

  Solve a question answering task with interleaving Thought, Action, Observation   usteps. Thought can reason about the current situation, and Action can be three types: