voxelgpt VS Interactive-LLM-Powered-NPCs

Sometimes, however, it’s nice to be able to create a custom user interface. Take VoxelGPT for instance: the chatbot-like interface (plus easy statefulness!) is really only possible when given its own devoted space within the FiftyOne App. In FiftyOne, creating custom interfaces like this is possible via JavaScript Plugins, which give you blank canvases on which to design workflows and experiences.

Over the past year, I’ve created a lot of GitHub repositories. Some of these repositories, such as my 10 Weeks of Plugins repo, are effectively aggregators — centralized locations that I can drive people to, almost like a personal Awesome list. Other repositories like VoxelGPT contain many moving pieces. Others still like Papers with Data are essentially minimum viable products for websites!

For all you developers, grab the code here and start exploring the future of AI: https://github.com/voxel51/voxelgpt

It is different this time, though. Take a look at this open source project.[1]

This is a system which lets you talk to NPCs in video games. It's a collection of off the shelf components held together by some Python code. The components do this:

- Listen to the user talking and convert speech to text.

- Watch the user's facial expressions via webcam.

- Watch the game, and use face recognition on the game images to determine what character is being addressed.

- Run the user's text through a LLM preloaded with about 30 lines of info about the NPC to generate a reply.

- Generate voice output in a voice generated to match the character's persona.

- Modify the image of the character on screen to animate their facial expressions to match the voice output. This is done on the output image, not by animating the 3D character.

Five years ago, that was science fiction. A year ago, half that stuff wouldn't work right. Now it's someone's hobby project.

[1] https://github.com/AkshitIreddy/Interactive-LLM-Powered-NPCs