openai-cookbook VS OpenCharacters

A pretrained LLM is a closed-book system: It can only access information that it was trained on. With domain fine-tuning, the system manifests additional material. An early prototype of this technique was shown in this OpenAi cookbook: For the target domain, text was embedded using an API, and then when using the LLM, embeddings were retrieved using semantic similarity search to formulate an answer. Although this approach evolved to retrieval-augmented generation, its still a technique to adapt a Gen2 (2020) or Gen3 (2022) LLM into a question-answering system.

https://github.com/openai/openai-cookbook/blob/main/examples...

It sounds like you are trying to tweak embeddings for similarity search. Rather than fine-tune the model's layers, you may want to try training a linear transformation the existing model's output embedding. Openai has a cookbook on how to do that. You will need some data though - but I think you can try it with ~20 pieces of synthetically generated data.

tutorial from OpenAI https://github.com/openai/openai-cookbook/blob/main/examples/Question_answering_using_embeddings.ipynb

OpenAI Cookbook

OpenAI community repo with lots of examples: https://github.com/openai/openai-cookbook

I also like Open Charactershttps://josephrocca.github.io/OpenCharacters/

I don't think he's using a 32k context. Rather, the OpenCharacter framework he's using can summarize and save memories locally for future reference.

For writing and specific intelligence tasks like programming or tax accounting, I use this Open source code that permanently aligns gpt4 and gives it semantic memory: https://josephrocca.github.io/OpenCharacters/

Consider this github project : https://github.com/josephrocca/OpenCharacters