cloudseeder VS promptfoo

Submission: This is a simple terminal for 'output/read-only' usage in a Fyne GoLang app. You can basically run a command and it will feel like its running in a terminal and the output will go to said terminal. This strips out all ansi, escape and control codes.

It's part of Cloud Seeder [1] which we wrote open source and in a way so that everyone can use the parts [2][3] easily.

[1] https://github.com/ipv6rslimited/cloudseeder - 1 click host with hands free upgrades your favorite self hosted appliance

[2] https://github.com/ipv6rslimited/configurator - make a config screen from a json

[3] https://github.com/ipv6rslimited/tray - make a system tray from a json

Ollama is really organized - it relies on llama but the UX and organization it provides makes it legit. We recently made a one-click wizard to run Open WebUI and Ollama together, self hosted and remotely accessible but locally hosted [1]

[1] https://github.com/ipv6rslimited/cloudseeder

Salutations HN,

We're excited to introduce Cloud Seeder, a tool designed to make self-hosting easy.

Problem Identification:

We realized there were many discussions about self-hosting but they never transitioned into action. After surveying potential users, two major obstacles emerged:

1. Lack of an external IP address.

2. The complexity of setting up and maintaining a server.

Our Solution - Cloud Seeder:

To make self-hosting straightforward, we developed Cloud Seeder. It's a platform that supports one-click deployment for over 20+ different appliances (and growing upon request), ranging from Mastodon (a federated social network that communicates with Meta's Threads) to Nextcloud (a robust, self-hosted alternative to major collaboration platforms).

Cloud Seeder is fully open source. The code is available on GitHub [1].

We believe Cloud Seeder will play a crucial role in fostering a more decentralized internet.

We hope you'll give this a try and start self hosting!

We also include a Configurator [2] and Tray [3] on github which lets you dynamically (from just a json) create dialog boxes and system trays. We use this in our application and thought it might be useful to people who don't want to spend time on GUI but would love to reach people with such UX.

[1] https://github.com/ipv6rslimited/cloudseeder

[2] https://github.com/ipv6rslimited/configurator

[3] https://github.com/ipv6rslimited/tray

Jumping in because I'm a big believer in (1) local LLMs, and (2) evals specific to individual use cases.

[0] https://github.com/typpo/promptfoo

Congrats on the launch!

I've been interested in automatic testset generation because I find that the chore of writing tests is one of the reasons people shy away from evals. Recently landed eval testset generation for promptfoo (https://github.com/typpo/promptfoo), but it is non-RAG so more simplistic than your implementation.

Was also eyeballing this paper https://arxiv.org/abs/2401.03038, which outlines a method for generating asserts from prompt version history that may also be useful for these eval tools.

Thanks for the promptfoo mention. For anyone else who might prefer deterministic, programmatic evaluation of LLM outputs, I've been building promptfoo: https://github.com/typpo/promptfoo

Example asserts include basic string checks, regex, is-json, cosine similarity, etc.