tvm VS cog

Yes. Web-llm is a wrapper of tvmjs: https://github.com/apache/tvm

Just wrappers all the way down

Yes, this is coming! Myself and others at OctoML and in the TVM community are actively working on multi-gpu support in the compiler and runtime. Here are some of the merged and active PRs on the multi-GPU (multi-device) roadmap:

Support in TVM’s graph IR (Relax) - https://github.com/apache/tvm/pull/15447

Would it make sense to have a backend support for OpenXLA, Apache TVM, Jittor or other similar to get free GPU, TPU and other accelerators for free ?

I have tried the iPhone app. It's fast. They're using Apache TVM which should allow better use of native accelerators on different devices. Like using metal on Apple and Vulcan or CUDA or whatever instead of just running the thing on the CPU like llama.cpp.

I was going to answer the same, I find the approach of machine learning compilers that directly compile models to host and device code better than having to bring a huge runtime. There are exciting projects in this area like TVM Unity, IREE [2], or torch.export [3]

[1] https://github.com/apache/tvm/tree/unity

[2] https://pytorch.org/get-started/pytorch-2.0/#inference-and-e...

[3] https://pytorch.org/get-started/pytorch-2.0/#inference-and-e...

Apache TVM home page: https://tvm.apache.org/

It's pretty clear its referring to the output of Apache TVM and Meta's Glow

As mentioned in sibling comments, Torch is indeed the glue in this implementation. Other glues are TVM[0] and ONNX[1]

These just cover the neural net though, and there is lots of surrounding code and pre-/post-processing that isn't covered by these systems.

For models on Replicate, we use Docker, packaged with Cog for this stuff.[2] Unfortunately Docker doesn't run natively on Mac, so if we want to use the Mac's GPU, we can't use Docker.

I wish there was a good container system for Mac. Even better if it were something that spanned both Mac and Linux. (Not as far-fetched as it seems... I used to work at Docker and spent a bit of time looking into this...)

[0] https://tvm.apache.org/

Or use TVM, the idea is to compile your model into code that you can load at runtime. Similar to onnxruntime, it only does DNN inference; so you need domain-specific code.

View on GitHub

I'm in the same situation. I found this cog project to dockerise ML https://github.com/replicate/cog : you write just one python class and a yaml file, and it takes care of the "CUDA hell" and deps. It even creates a flask app in front of your model.

That helps keep your system clean, but someone with big $s please rewrite pytorch to golang or rust or even nodejs / typescript.

https://github.com/replicate/cog

Our thinking was just that a bunch of folks will want to fine-tune right away, then deploy the fine-tunes, so trying to make that easy... Or even just deploy the models-as-is on their own infra without dealing with CUDA insanity!

I have used this tool before: https://github.com/replicate/cog/tree/main

Then I discovered Cog: Containers for Machine Learning. Looks like a way more flexible solution to plug in the existing infrastructure: you write your custom code and Cog plugs it in a Docker image with FastAPI, no extra ecosystem complexity added.

Highly recommend using cog https://github.com/replicate/cog

Dokku Maintainer here.

Dokku also supports Dockerfiles, Docker Images, Tarballs (similar to heroku slugs), and Cloud Native Buildpacks. I'm also actively working on AWS Lambda support (both for simple usage without much config as well as SAM-based usage) and investigating Replicate's Cog[1] and Railways Nixpacks[2] functionalities for building apps.

There are quite a few options in the OSS space (as well as Commercial offerings from new startups and popular incumbents). It's an interesting space to be in, and its always fun to see how new offerings innovate on existing solutions.

  [1] https://github.com/replicate/cog