aici VS Awesome-LLM-Productization

Have you looked at AICI by Microsoft yet?

https://github.com/microsoft/aici/

I have been planning to work on something like this. I think that eventually, someone will crack the "binary in -> good source code out of LLM" pipeline but we are probably a few years away from that still. I say a few years because I don't think there's a huge pile of money sitting at the end of this problem, but maybe I'm wrong.

A really good "stop-gap" approach would be to build a decompilation pipeline using Ghidra in headless mode and then combine the strict syntax correctness of a decompiler with the "intuition/system 1 skills" of an LLM. My inspiration for this setup comes from two recent advancements, both shared here on HN:

1. AlphaGeometry: The Decompiler and the LLM should complement each other, covering each other's weaknesses. https://deepmind.google/discover/blog/alphageometry-an-olymp...

2. AICI: We need a better way of "hacking" on top of these models, and being able to use something like AICI as the "glue" to coordinate the generation of C source. I don't really want the weights of my LLM to be used to generate syntactically correct C source, I want the LLM to think in terms of variable names, "snippet patterns" and architectural choices while other tools (Ghidra, LLVM) worry about the rest. https://github.com/microsoft/aici

Obviously this is all hand-wavey armchair commentary from a former grad student who just thinks this stuff is cool. Huge props to these researchers for diving into this. I know the authors already mentioned incorporating Ghidra into their future work, so I know they're on the right track.

We believe Guidance can run on top of AICI (we're working on efficient Earley parser for that [0], together with local Guidance folks). AICI is generally lower level (though our sample controllers are at similar level to Guidance).

[0] https://github.com/microsoft/aici/blob/main/controllers/aici...

There are a bunch of open source projects or commercial projects productising LLMs, but there are still challenges in, e.g., latency, cost reduction, fine-tuning, data preparation, monitoring to name a few.

This repo monitors projects or packages that can help you speed up the adoption, with boilerplate, E2E backend and real-world use cases as its goal.

Please feel free to open issues and more contents will be coming soon.

https://github.com/oscinis-com/Awesome-LLM-Productization/