tvm VS PhysX-3.4

Yes. Web-llm is a wrapper of tvmjs: https://github.com/apache/tvm

Just wrappers all the way down

Yes, this is coming! Myself and others at OctoML and in the TVM community are actively working on multi-gpu support in the compiler and runtime. Here are some of the merged and active PRs on the multi-GPU (multi-device) roadmap:

Support in TVM’s graph IR (Relax) - https://github.com/apache/tvm/pull/15447

Would it make sense to have a backend support for OpenXLA, Apache TVM, Jittor or other similar to get free GPU, TPU and other accelerators for free ?

I have tried the iPhone app. It's fast. They're using Apache TVM which should allow better use of native accelerators on different devices. Like using metal on Apple and Vulcan or CUDA or whatever instead of just running the thing on the CPU like llama.cpp.

I was going to answer the same, I find the approach of machine learning compilers that directly compile models to host and device code better than having to bring a huge runtime. There are exciting projects in this area like TVM Unity, IREE [2], or torch.export [3]

[1] https://github.com/apache/tvm/tree/unity

[2] https://pytorch.org/get-started/pytorch-2.0/#inference-and-e...

[3] https://pytorch.org/get-started/pytorch-2.0/#inference-and-e...

Apache TVM home page: https://tvm.apache.org/

It's pretty clear its referring to the output of Apache TVM and Meta's Glow

As mentioned in sibling comments, Torch is indeed the glue in this implementation. Other glues are TVM[0] and ONNX[1]

These just cover the neural net though, and there is lots of surrounding code and pre-/post-processing that isn't covered by these systems.

For models on Replicate, we use Docker, packaged with Cog for this stuff.[2] Unfortunately Docker doesn't run natively on Mac, so if we want to use the Mac's GPU, we can't use Docker.

I wish there was a good container system for Mac. Even better if it were something that spanned both Mac and Linux. (Not as far-fetched as it seems... I used to work at Docker and spent a bit of time looking into this...)

[0] https://tvm.apache.org/

Or use TVM, the idea is to compile your model into code that you can load at runtime. Similar to onnxruntime, it only does DNN inference; so you need domain-specific code.

Depends if the drivers you have already have the GPU drivers already but if you want a clean slate you can use DDU first then install fresh from www.nvidia.com on the latest driver

Then, reboot and head to www.nvidia.com to download the latest drivers for your GPU.

Try it manually from www.nvidia.com, the file is at least 900mb so if you have a couple gigabytes free it should be fine

Setting a new benchmark for the future of automobiles, industry leaders MediaTek and NVIDIA are joining forces to create a cutting-edge artificial intelligence (AI) and accelerated computing experience for the automotive industry. From the entry-level to premium, this partnership signifies an evolutionary leap for in-vehicle connectivity and infotainment solutions.

once it gets to the point where I enter nvidiia-smi I getMNVIDIA-SMI has failed because it couldn't communicate with the NVIDIA driver. Make sure that the latest NVIDIA driver is installed and running. I downloaded the driver and had to run it in root where I was told I had X runnning and had to turn it off and to visit www.nvidia.com but there is no help there. I was also told to see /var/log/nvidia-installer.log for details. Any ideas?

Display driver uninstaller, remove all Nvidia GPU drivers and clean install new ones from www.nvidia.com , that will fix it.

Just go to www.nvidia.com and just install them manually

just download it from www.nvidia.com and worry about it next time

Run the program and remove and Nvidia GPU drivers that it finds. Grab the latest drivers from www.nvidia.com and try to install once DDU has removed all of the old ones. Make sure you restart after it's done.