YOLO-World VS plotly.js

2024 is shaping up to be the year of multimodal machine learning. From real-time text-to-image models and open-world vocabulary models to multimodal large language models like GPT-4V and Gemini Pro Vision, AI is primed for an unprecedented array of interactive multimodal applications and experiences.

Post author here. I like this idea. I plan to explore it and make a more generic solution. I'd love to have a point-and-click interface for annotating scenes.

For example, I'd like to be able to click on pieces of coffee equipment in a photo of my coffee setup so I can add sticky note annotations when you hover over each item.

For the bookshelves idea specifically, I would love to have a correction system in place. The problem isn't so much SAM as it is Grounding DINO, the model I'm using for object identification. I then pass each identified region to SAM and map the segmentation mask to the box.

Grounding DINO detects a lot of book spines, but often misses 1-2. I am planning to try out YOLO-World (https://github.com/AILab-CVC/YOLO-World), which, in my limited testing, performs better for this task.

Plotly is based on D3. Has both open-source version and paid option.

https://plotly.com/javascript/

Plotly as our chart renderer

In this lesson, we will go through the steps of using Plotly.js in React. Plotly.js is an Open Source Graphing Library. At the end of this lesson, you should be able to do a simple plot with Plotly.js.

Well, MathML[1] support is (nearly) everywhere now, and as the docs say:

MathML Core is a subset with increased implementation details based on rules from LaTeX and the Open Font Format. It is tailored for browsers and designed specifically to work well with other web standards including HTML, CSS, DOM, JavaScript.

I don't have a lot of experience working with this stuff (yet) but if you can script your MathML objects with Javascript, you should be able to make whatever interactive "stuff" you want in terms of math notation. Now drawing objects and plots and stuff is a different question.

There's stuff like Plotly[2], D3[3], Sigma[4], etc. that might be useful depending on exactly what effects you're going for.

[1]: https://developer.mozilla.org/en-US/docs/Web/MathML

[2]: https://github.com/plotly/plotly.js

[3]: https://d3js.org/

[4]: https://www.sigmajs.org/

This is the end of the lesson. For more information on Plotly.js, check the documentation Plotly.