LLMTest_NeedleInAHaystack VS gpt-pilot

This research follows the “haystack test” Greg Kamradt published when the update GPT-4 came out (twitter, code). That test provided useful insight into (the lack of) context recall performance. But it was performed on a very small sample test (limiting its statistical significance) and was initially limited to GPT-4 (he has since published an updated version that also uses Claude 2.1). Moreover, the test data consists of essays that were likely already used pretraining LLMs, and the results were evaluated by GPT-4, potentially introducing confounding variables into the mix.

For the past 6 months, I’ve been working on GPT Pilot (https://github.com/Pythagora-io/gpt-pilot) to understand how much we can really automate coding with AI, so I wanted to share our learnings so far and how far it’s able to go.

I’m working on an AI dev tool GPT Pilot that uses LLMs a lot. So, I was interested in context recall - however, it becomes more apparent at larger context sizes. In other words, how well can the LLM find the information it needs that is in the context? Less than ideal, as it turns out.

Learn more about it and give it a try here: https://github.com/Pythagora-io/gpt-pilot

I opened my account yesterday, as I wanted to test gpt-pilot, I generated an API key, added it to my .env file and made a small calculator just to see if it works (and it does). Today I got an email saying that my account was terminated for violating Terms of Service or not being in a supported country. The closest thing that I can think of for my ban after reading ToS and the list of countries was that my school uses GlobalProtect (a VPN) to connect to their services, so maybe that raised a red flag, but that's about it. I'm trying to find the option to "appeal the ban" on help.openai.com but I can't find it anywhere.