puter VS LAVIS

Puter is a cloud personal computer, and you can have all your files, apps, and games in one place accessible from anywhere at any time. So, basically a computer in your browser,

> it is almost impossible not to use any of those frameworks.

Agreed, but because it’s a lot easier to just search for “React programmer” these days than it is to evaluate lots of JavaScript candidates, which has a much wider scope and proficiency level. And they’ll fit right in when hired.

But not because it’s not inherently scalable. See puter[1] for instance, a fairly complex, 100k+ lines of code of direct DOM manipulation (jQuery, IIRC)

https://github.com/HeyPuter/puter

For multimodal, there's China (*many), then Salesforce.

Been trying for all day to get a working inference for this example: https://github.com/salesforce/LAVIS/tree/main/projects/instructblip

There's also been big breakthroughs in computer vision. Not that long ago it was hard to recognize if a photo contained a bird; that's solved now by models like CLIP, Yolo, or Segment Anything. Now research has moved on to generating 3D scenes from images or interactively answering questions about images.

The new hotness is multimodal vision-language models like InstructBLIP that can interactively answer questions about images. Check out the examples in the github repo, I would not have thought this was possible a few years ago.

Salesforce’s BLIP family has a new member– InstructBLIP, a vision-language instruction-tuning framework using BLIP-2 models. It has achieved state-of-the-art zero-shot generalization performance on a wide range of vision-language tasks, substantially outperforming BLIP-2 and Flamingo. (Source)

Github

Most of my workflows are self-made. For captioning I used Blip-2 in a custom script I made that automates the process by going into directories and their sub-directories and creates a .txt file beside each image. This way I can keep my images organized in their proper directories, without having to put dump them all in a single place.