tokencost VS llm_utils

Library for Estimating Token Costs

I really appreciate your engagement here and think it has great value on a personal level, but the length and claims tend to hide two very obvious, straightforward things:

1. They only support GPT3.5 and GPT4.0. Note here: [1], and that gpt-4o would get swallowed into gpt-4-0613.

2. This will lead to massive, significant, embarrassingly large error in calculations. Tokenizers are not mostly the same, within 10% error.

1. responsive to ex. "It's not just C100K though. It is for a few models [0]",

The link is to Tiktoken, OpenAI's tokenization library. There are literally more than GPT3.5 and GPT4.0 there, but they're just OpenAI's models, no one else's, none of the others in the long list in their documentation, and certainly not 400.

every single one of them is for a deprecated model, not served anymore, except c100k and o200k. As described above and shown in [1], their own code kneecaps the o200k and will use c100k"

2. Let me know what you'd want to see if you're curious the 30%+ error thing. I don't want to go to the trouble to guess at a test suite, then run one, that would make you confident you need to revise a prior that there's only +/- 10% difference between arbitrary tokenizers. I will almost assuredly choose one that isn't comprehensive enough, with your input.

For context, I run about 20 unit tests, for each of the big 5 providers, with the same prompts, to capture their input and output token counts to make sure I'm billing accurately.

Just to save you time, you won't be able to talk me down to "eh, good enough!" --- It *matters*, if it didn't, they'd be much more up front about the truth. Every single sign around the library is absolutely damning, and triangulates somewhere between lying and naivete. From the marketing claiming 400+, to the complete lack of note of these extreme* caveats in any documentation, the only thing being what I understand is a warning log.

[1] https://github.com/AgentOps-AI/tokencost/blob/e1d52dbaa3ada2...

> tiktoken.encoding_for_model(model)

Calling this where model == 'gpt-4o' will encode with CL200k no?

But yes, I do agree with you. I had time implementing non-tiktoken tokenizers for my project. I ended up manually adding tokenizer.json files into my repo.[1] The other options is downloading from HF, but the official repos where the model's tokenizer.json lives require agreeing to their terms to access. So it requires an HF key, and agreeing to the terms. So not a good experience for a consumer of the package.

> Message frame tokens?

Do you mean the chat template tokens? Oh, that's another good point. Yeah, it counts OpenAI prompt tokens. I solved this by implementing a Jinja templating engine to create the full prompt. [2] Granted, both llama.cpp and mistral-rs do this on the backend, so it's purely for counting tokens. I guess it would make sense to add a function to convert tokens to Dollars.

[1] https://github.com/ShelbyJenkins/llm_utils/tree/main/src/mod...