tokencost VS litellm

Library for Estimating Token Costs

I really appreciate your engagement here and think it has great value on a personal level, but the length and claims tend to hide two very obvious, straightforward things:

1. They only support GPT3.5 and GPT4.0. Note here: [1], and that gpt-4o would get swallowed into gpt-4-0613.

2. This will lead to massive, significant, embarrassingly large error in calculations. Tokenizers are not mostly the same, within 10% error.

1. responsive to ex. "It's not just C100K though. It is for a few models [0]",

The link is to Tiktoken, OpenAI's tokenization library. There are literally more than GPT3.5 and GPT4.0 there, but they're just OpenAI's models, no one else's, none of the others in the long list in their documentation, and certainly not 400.

every single one of them is for a deprecated model, not served anymore, except c100k and o200k. As described above and shown in [1], their own code kneecaps the o200k and will use c100k"

2. Let me know what you'd want to see if you're curious the 30%+ error thing. I don't want to go to the trouble to guess at a test suite, then run one, that would make you confident you need to revise a prior that there's only +/- 10% difference between arbitrary tokenizers. I will almost assuredly choose one that isn't comprehensive enough, with your input.

For context, I run about 20 unit tests, for each of the big 5 providers, with the same prompts, to capture their input and output token counts to make sure I'm billing accurately.

Just to save you time, you won't be able to talk me down to "eh, good enough!" --- It *matters*, if it didn't, they'd be much more up front about the truth. Every single sign around the library is absolutely damning, and triangulates somewhere between lying and naivete. From the marketing claiming 400+, to the complete lack of note of these extreme* caveats in any documentation, the only thing being what I understand is a warning log.

[1] https://github.com/AgentOps-AI/tokencost/blob/e1d52dbaa3ada2...

Very cool! Is this cost directory you're using the best source for historical cost per 1M tokens? https://github.com/BerriAI/litellm/blob/main/model_prices_an...

Next, we'll show how an Embeddings database can integrate with external API services via LiteLLM .

There are a few libs that already abstract this away, for example:

- https://github.com/BerriAI/litellm

- https://jxnl.github.io/instructor/

- langchain

It's not hard for me to imagine a future where there is something like the CNCF for AI models, tools, and infra.

Hey, are you just looking for litellm - https://github.com/BerriAI/litellm

context - i'm the repo maintainer

It should be possible using LiteLLM and a patch or a proxy.

https://github.com/BerriAI/litellm

This article will demonstrate how txtai can integrate with llama.cpp, LiteLLM and custom generation methods. For custom generation, we'll show how to run inference with a Mamba model.

I use this with ollama and works perfectly https://github.com/BerriAI/litellm

Another abstraction layer library is: https://github.com/BerriAI/litellm

For me the killer feature of a library like this would be if it implemented function calling. Even if it was for a very restricted grammar - like the traditional ReAct prompt:

  Solve a question answering task with interleaving Thought, Action, Observation   usteps. Thought can reason about the current situation, and Action can be three types: