promptfoo VS OLMo

Jumping in because I'm a big believer in (1) local LLMs, and (2) evals specific to individual use cases.

[0] https://github.com/typpo/promptfoo

Congrats on the launch!

I've been interested in automatic testset generation because I find that the chore of writing tests is one of the reasons people shy away from evals. Recently landed eval testset generation for promptfoo (https://github.com/typpo/promptfoo), but it is non-RAG so more simplistic than your implementation.

Was also eyeballing this paper https://arxiv.org/abs/2401.03038, which outlines a method for generating asserts from prompt version history that may also be useful for these eval tools.

Thanks for the promptfoo mention. For anyone else who might prefer deterministic, programmatic evaluation of LLM outputs, I've been building promptfoo: https://github.com/typpo/promptfoo

Example asserts include basic string checks, regex, is-json, cosine similarity, etc.

Olmo from AI2. They released the model weights plus training data and training code.

link: https://allenai.org/olmo

It looks like the weights [0] and code [1] are Apache licensed, but the training data [2] is using the license that OP is quoting from.

[0] https://huggingface.co/allenai/OLMo-7B

[1] https://github.com/allenai/OLMo

[2] https://huggingface.co/datasets/allenai/dolma