[P] 4.5 times faster Hugging Face transformer inference by modifying some Python AST

• transformer-deploy

  8 1,626 6.8 Python

  Efficient, scalable and enterprise-grade CPU/GPU inference server for 🤗 Hugging Face transformer models 🚀

  

Regarding CPU inference, quantization is very easy, and supported by Transformer-deploy , however performance on transformer are very low outside corner cases (like no batch, very short sequence and distilled model), and last Intel generation CPU based instance like C6 or M6 on AWS are quite expensive compared to a cheap GPU like Nvidia T4, to say it otherwise, on transformer, until you are ok with slow inference and takes a small instance (for a PoC for instance), CPU inference is probably not a good idea.

• sparsednn

  1 51 0.0 Python

  Fast sparse deep learning on CPUs

If you are ever interested in looking at pruning, happy to integrate my open source library https://github.com/marsupialtail/sparsednn. Latest update has unstructured and structured sparse int8 kernels. 3x speedup over dense int8 at 90 percent sparsity with 1x4 blocks.

• Scout Monitoring

  www.scoutapm.com featured

  Free Django app performance insights with Scout Monitoring. Get Scout setup in minutes, and let us sweat the small stuff. A couple lines in settings.py is all you need to start monitoring your apps. Sign up for our free tier today.

  

• TensorRT

  5 2,373 9.5 Python

  PyTorch/TorchScript/FX compiler for NVIDIA GPUs using TensorRT (by pytorch)

  

Have you tried the new Torch-TensorRT compiler from NVIDIA?

• mmrazor

  4 1,387 2.8 Python

  OpenMMLab Model Compression Toolbox and Benchmark.

  

https://github.com/open-mmlab/mmrazor ,it may work for you~

Suggest a related project