-
> Tangential critiques are preferable?
Not at all. Reasoned arguments and adding information (the examples I gave were what I thought your main points were) to the discussion are preferable to (what seemed to be) character attacks. Your comment here, as an example, was mostly great. It provides the same level of usefulness to anyone reading it (highlighting that the computation is just C100K and that people will be misled if they try to use it the wrong way), and you also added reasoned counter-arguments to my OOM idea and several other interesting pieces of information. To the extent that you kept the character attacks against the author, you at least softened the language.
Respectfully attacking ideas instead of people is especially important in online discourse like this. Even if you're right, attacking people tends to spiral a conversation out of control and convince no one (often persuading them of the opposite of whatever you were trying to say).
> just C100K
It's not just C100K though. It is for a few models [0], but even then the author does warn the caller (mind you, I prefer mechanisms like an `allow_approximate_token_count=False` parameter or whatever, but that's not fraud on the author's part; that's a dangerous API design).
Going back to the "tone" thing, calling out those sorts of deficiencies is a great way to warn other people, let them decide if that sort of thing matters for their use case, point out potential flaws in your own reasoning (e.g., it's not totally clear to me if you think the code always uses C100K or always uses it for a subset of models, but if it's the former then you'd probably be interested in knowing that the tokenizer is actually correct for most models) and discuss better API designs. It makes everyone better off for having read your comment and invites more discussions which will hopefully also make everyone better off.
> outside the realm of even needing a function
Maybe! I'd argue that it's useful to have all those prices (especially since not all tokens are created equally) in one place somewhere, but arguing that this is left-pad for LLM pricing is also a reasonable thing to talk about.
> it's impossible to get definitive error bars
That's also true, but that doesn't matter for every application. E.g., suppose you want to run some process on your entire corporate knowledge-base and want a ballpark estimate of costs. The tokenizer error is on average much smaller than the 30%+ you saw for some specific (currently unknown to us here at HN) very small input. Just run your data through this tool, tally up the costs, and you ought to be within 10%. Nobody cares if it's a $900 project or a $1300 project (since nobody is allocating expensive, notoriously unpredictable developers to a project with only 10-30% margins). You just tell the stakeholders it'll cost $2k and a dev-week, and if it takes less then everyone is happily surprised. If they say no at that estimate, they probably wouldn't have been ecstatic with the result if it actually cost $900 and a dev-day anyway.
[0] https://github.com/AgentOps-AI/tokencost/blob/main/tokencost...
-
Scout Monitoring
Free Django app performance insights with Scout Monitoring. Get Scout setup in minutes, and let us sweat the small stuff. A couple lines in settings.py is all you need to start monitoring your apps. Sign up for our free tier today.
-
openai-messages-token-helper
A utility library for dealing with token counting for messages sent to an LLM (currently OpenAI models only)
I grappled with that issue for https://github.com/pamelafox/openai-messages-token-helper as I wanted to be able to use it for a quick token check with SLMs as well, so I ended up adding a parameter "fallback_to_default" for developers to indicate they're okay with assuming gpt-35 BPE encoding.
-
-
> tiktoken.encoding_for_model(model)
Calling this where model == 'gpt-4o' will encode with CL200k no?
But yes, I do agree with you. I had time implementing non-tiktoken tokenizers for my project. I ended up manually adding tokenizer.json files into my repo.[1] The other options is downloading from HF, but the official repos where the model's tokenizer.json lives require agreeing to their terms to access. So it requires an HF key, and agreeing to the terms. So not a good experience for a consumer of the package.
> Message frame tokens?
Do you mean the chat template tokens? Oh, that's another good point. Yeah, it counts OpenAI prompt tokens. I solved this by implementing a Jinja templating engine to create the full prompt. [2] Granted, both llama.cpp and mistral-rs do this on the backend, so it's purely for counting tokens. I guess it would make sense to add a function to convert tokens to Dollars.
[1] https://github.com/ShelbyJenkins/llm_utils/tree/main/src/mod...
-
Anthropic actually has a Claude 3 tokenizer tucked away in one of their repos: https://github.com/anthropics/anthropic-tokenizer-typescript
At this moment, Tokencost uses the OpenAI tokenizer as a default tokenizer, but this would be a welcome PR!
-
litellm
Call all LLM APIs using the OpenAI format. Use Bedrock, Azure, OpenAI, Cohere, Anthropic, Ollama, Sagemaker, HuggingFace, Replicate (100+ LLMs)
Very cool! Is this cost directory you're using the best source for historical cost per 1M tokens? https://github.com/BerriAI/litellm/blob/main/model_prices_an...
