simple-proxy-for-tavern VS Local-LLM-Comparison-Colab-UI

Been using 1.9 for a while now, but heard about all the nice new extras in 1.1 so I decided to upgrade today. Silly me! Well ST Itself is running fine I've got that up no issues, extras is installed and koboldcpp is up to date...but it won't connect ST to KCPP. That being https://github.com/anon998/simple-proxy-for-tavern. It's worked like a dream for weeks up until today when I went to update and now the two can't talk. I did a fresh git pull of a new version - nothing. Rolled back to 1.9.7 - and it works like a charm. Anyone else had these issues?

https://github.com/anon998/simple-proxy-for-tavern/issues/25 It threw this error at me for a moment when I tried open AI, but not when running my Local LLM Via Kobold.

Still, nothing beats the SillyTavern + simple-proxy-for-tavern setup for me. But currently there's even a known issue with that and koboldcpp regarding sampler order used in the proxy presets (PR for fix is waiting to be merged, until it's merged, manually changing the presets may be required).

I’m using this https://github.com/anon998/simple-proxy-for-tavern

SillyTavern is just a frontend so it's as fast or slow as your backend. With simple-proxy-for-tavern you can use llama.cpp directly, no Python involved, so SillyTavern will be as fast as llama.cpp itself.

I also use the simple-proxy-for-tavern in between koboldcpp and the frontend, which does some magic behind the scenes to improve things further. Takes some time to understand and configure it all, but once done, it's definitely worth the effort, as nothing beats that setup for roleplaying and chat.

Really? Then we definitely have different experiences (or different ways to interact) with Guanaco. It's been the most unrestricted model I've tried, and I tried them all, but I'm using SillyTavern and the simple-proxy-for-tavern which combined with a little prompting liberates basically any model.

Why reinvent the wheel? Just use SillyTavern, ideally with the simple-proxy-for-tavern. That does it all, and more.

In terms of speed, we're talking about 140t/s for 7B models, and 40t/s for 33B models on a 3090/4090 now.[1] (1 token ~= 0.75 word) It's quite zippy. llama.cpp performs close on Nvidia GPUs now (but they don't have a handy chart) and you can get decent performance on 13B models on M1/M2 Macs.

You can take a look at a list of evals here: https://llm-tracker.info/books/evals/page/list-of-evals - for general usage, I think home-rolled evals like llm-jeopardy [2] and local-llm-comparison [3] by hobbyists are more useful than most of the benchmark rankings.

That being said, personally I mostly use GPT-4 for code assistance to that's what I'm most interested in, and the latest code assistants are scoring quite well: https://github.com/abacaj/code-eval - a recent replit-3b fine tune the human-eval results for open models (as a point of reference, GPT-3.5 gets 60.4 on pass@1 and 68.9 on pass@10 [4]) - I've only just started playing around with it since replit model tooling is not as good as llamas (doc here: https://llm-tracker.info/books/howto-guides/page/replit-mode...).

I'm interested in potentially applying reflexion or some of the other techniques that have been tried to even further increase coding abilities. (InterCode in particular has caught my eye https://intercode-benchmark.github.io/)

[1] https://github.com/turboderp/exllama#results-so-far

[2] https://github.com/aigoopy/llm-jeopardy

[3] https://github.com/Troyanovsky/Local-LLM-comparison/tree/mai...

[4] https://github.com/nlpxucan/WizardLM/tree/main/WizardCoder

The best 7B I tried is WizardLM. It's my go-to model.

If you want to try it out, you can use Google Colab here with Oobabooga Text Generation UI: Link (Remember to check the instruction template and generation parameters)

I'm really impressed by wizardLM-7B.q4_2 (GPT4all) running on my 8gb M2 Mac Air. Fast response, fewer hallucinations than other 7B models I've tried. GPT4All's beta document collection and query function is respectable--going to test it more tomorrow. FWIW wizardLM-7B.q4_2 was ranked very high here https://github.com/Troyanovsky/Local-LLM-comparison.

I made a series of Colab notebooks for different models: https://github.com/Troyanovsky/Local-LLM-comparison

I found it performing very well too in my testing (Repo). It's my second favorite model after WizardLM-13B.

My entire list at: Local LLM Comparison Repo

I just tried HyperMantis and updated the results in the repo. It performs not bad but worse than Nous-Hermes-13B.