Blog
deployment

Vulkan Support: LLMs for Everyone

June Marcuse
May 1, 2024
•
2
-minute read

It has long been the case that machine learning models are run on the GPU to improve their performance. The GPU is far more effective at the kinds of computations needed for AI than the CPU, and so GPU compute libraries such as Cuda and ROCm are typically used.

However, requiring the support of these libraries can restrict which graphics cards are compatible, leaving many with older or less popular cards unable to run LLMs efficiently.

Tabby is happy to announce that we now support Vulkan, a graphics library created primarily for games. Its original purpose means that it is designed to work on a very broad range of cards, and leveraging it to host LLMs means that we can now offer GPU acceleration to people whose cards are not supported by Cuda and ROCm.

Vulkan works on basically any GPU, so if you have previously been forced to host local models on your CPU, now is the time to see what Tabby with Vulkan can do for you!

Vulkan Installation​

To begin, first make sure that you have Vulkan installed.

For Windows users, Vulkan may be natively supported. Otherwise, the Vulkan SDK can be downloaded at https://vulkan.lunarg.com/sdk/home#windows.

For Linux users, Vulkan can be installed through your package manager:

  • Arch Linux: vulkan-icd-loader (universal), and also install vulkan-radeon (for AMD) or vulkan-nouveau (for Nvidia)
  • Debian Linux: libvulkan1

Tabby Installation​

To start using Tabby with Vulkan, first download one of the pre-built Vulkan binaries for your platform:

Running​

Once you've installed the appropriate binary, you can simply run it from the command line:

For Windows, open a command prompt and navigate to the download folder, then run:

tabby_x86_64-windows-msvc-vulkan serve --model StarCoder-1B --device vulkan

For Linux:

./tabby_x86_64-manylinux2014-vulkan serve --model StarCoder-1B --device vulkan

When it starts, you should see a printout indicating that Vulkan has found your card and is working properly:

Now enjoy your speedy completions!

Share this post
deployment

Stay Updated with Tabby News

Subscribe to our newsletter for the latest updates and news about Tabby.

By joining, you agree to our Terms and Conditions.
Thank you! We've received your submission.
Oops! Something went wrong. Please try again.
Deploying a Tabby Instance in Hugging Face Spaces
September 5, 2023
  ▪︎  
4
min read
deployment
Running Tabby Locally with AMD ROCm
January 24, 2024
  ▪︎  
2
min read
deployment
Deploy Tabby in Air-Gapped Environment with Docker
March 25, 2024
  ▪︎  
2
min read
deployment
[ View our Full blog ]
Discover Tabby Unlock Your Coding Potential
Explore the Power of Tabby, the Self-Hosted AI Coding Assistant

  
                                                                                                             
                                                                                                             
                                                                                                             
                                                                                                             
333                                                                            333333                        
444   7                                                                       66466                          
00   313333                                                                 0000                             
   55555                                                                                                  331
  666                                                                                                    444 
888       777777                                                                                        888  
0       3311                                                                                            0    
    222222                                                                                                   
  455555         77777777                                                                                    
 666664       1111117                                                                                        
999999     3333333                                                                                    7      
8888     2222222   77777777                                                                    777           
000    5555555   1111111                                                                     33333           
0   4444444    1111113                                                                     55555             
  6666666    2333333                                                                      66664              
 999999    2222222                           7                                           8888                
888888    555555      77777777     77777777                                              00                  
0000   44444444     77777777    177777777                                                                    
00  4444444446   111111111    11111111                                                                       
0  666666666   13313131    3331313        777777                                                             
 999999999    333333     3333333        777777                                                               
8888888      222223    2222222        1111117                                                                
0000       222222     2222225      11111111                                                                  
000     55555555    5555555     333333333    7                                                               
0      5444444   544444445    3333322                                                                        
      444444   4444444444  2222222                                                             7             
      6666  66666666666 5552522   7 777   777 77    7 7   7                                  7               
     6666 66666666666  55555  7777777777  7777     7777   7777       7 7777       7 77 77 7      7 7   777   
   999999999999999    4444  7777777777 777777    177777  777777     777777  7777777777        7777 7777777   
 88898889888898     4444 111111111   111111     1111111  77777    777777 1777777777         717777777777     
88888888888        666 11111111    111111     11111111  11111    711111111111111          1111111111111      
0000000          666 3131313     3131313     11313133  11113    111111111111           111111111111111       
00000         9999  333333      3333333    33333333   13333   3333333333         333333333333333311          
000        999999  33333      3333333     333333     3333    3333333          33333333333333                 
0       8888888  222222    222222222    222222     3232    32332            3323232323223                    
    88888880    22222    222222222   2222222     2222    2222             2222222222222                      
 00000000     555555    55555555  255555552    2222    22222            222222222222                         
000000       55555    55555555 5555555555    5555   255555            5555555555552                          
0000       555555    555555555555555554    5555   55555555          555555555555                             
00      4444444     44444444444444445    44444 4444444445         44444444445                                
      44444444     444444444444444      444444444444 4444       4444444444                                   
    46666664      66666666666664       44444444644   444      46444444                                       
 66666666        6666666666666        666666666    6666      666666                                          
6969666        66969696969696       96666666     66666      66666     777777777    777777                    
99999        99999999999999       9999999      999999      99999     111111113    11111                      
99         999999999999        9999999        999999      99999    333333333     33333                       
         888888888      9988888888898       8888899     888889    22222222      22222   77777777             
      888888888      88888888888888      8888888     88888888   55555555       5555    111111                
    088888888      0888888888888       088888      88888888   444444444      44444    22222                  
  000000000      00000000000        000000      0000000000  666666666      66666    55555      1111111       
0000000000     000000000     00000000000    000000000000   99999999      999999    66666     2222222         
0 000000    000000000   0000000000000    0000000000000    8888888      888888   9999999    4444444           
000000    00000000    0000000000000   00000000000000     00000      0000000   0000000    89999998         7  
000    00000000     0000000000000   000000000000000    00000       000000   00000000    000000   9999999     
  


Get Started with our Community Plan Today
Get Started
Simple self-onboarding
Free community plan
Local-first deployment




  
333                                                                            333333                        
444   7                                                                       66466                          
00   313333                                                                 0000                             
   55555                                                                                                  331
  666                                                                                                    444 
888       777777                                                                                        888  
0       3311                                                                                            0    
    222222                                                                                                   
  455555         77777777                                                                                    
 666664       1111117                                                                                        
999999     3333333                                                                                    7      
8888     2222222   77777777                                                                    777           
000    5555555   1111111                                                                     33333           
0   4444444    1111113                                                                     55555             
  6666666    2333333                                                                      66664              
 999999    2222222                           7                                           8888                
888888    555555      77777777     77777777                                              00                  
0000   44444444     77777777    177777777                                                                    
00  4444444446   111111111    11111111                                                                       
0  666666666   13313131    3331313        777777                                                             
 999999999    333333     3333333        777777                                                               
8888888      222223    2222222        1111117                                                                
0000       222222     2222225      11111111                                                                  
000     55555555    5555555     333333333    7                                                               
0      5444444   544444445    3333322                                                                        
      444444   4444444444  2222222                                                             7             
      6666  66666666666 5552522   7 777   777 77    7 7   7                                  7               
     6666 66666666666  55555  7777777777  7777     7777   7777       7 7777       7 77 77 7      7 7   777   
   999999999999999    4444  7777777777 777777    177777  777777     777777  7777777777        7777 7777777   
 88898889888898     4444 111111111   111111     1111111  77777    777777 1777777777         717777777777     
88888888888        666 11111111    111111     11111111  11111    711111111111111          1111111111111      
0000000          666 3131313     3131313     11313133  11113    111111111111           111111111111111       
00000         9999  333333      3333333    33333333   13333   3333333333         333333333333333311          
000        999999  33333      3333333     333333     3333    3333333          33333333333333                 
0       8888888  222222    222222222    222222     3232    32332            3323232323223                    
    88888880    22222    222222222   2222222     2222    2222             2222222222222                      
 00000000     555555    55555555  255555552    2222    22222            222222222222                         
000000       55555    55555555 5555555555    5555   255555            5555555555552                          
0000       555555    555555555555555554    5555   55555555          555555555555                             
00      4444444     44444444444444445    44444 4444444445         44444444445                                
      44444444     444444444444444      444444444444 4444       4444444444                                   
    46666664      66666666666664       44444444644   444      46444444                                       
 66666666        6666666666666        666666666    6666      666666                                          
6969666        66969696969696       96666666     66666      66666     777777777    777777                    
99999        99999999999999       9999999      999999      99999     111111113    11111                      
99         999999999999        9999999        999999      99999    333333333     33333                       
         888888888      9988888888898       8888899     888889    22222222      22222   77777777             
      888888888      88888888888888      8888888     88888888   55555555       5555    111111                
    088888888      0888888888888       088888      88888888   444444444      44444    22222                  
  000000000      00000000000        000000      0000000000  666666666      66666    55555      1111111       
0000000000     000000000     00000000000    000000000000   99999999      999999    66666     2222222         
0 000000    000000000   0000000000000    0000000000000    8888888      888888   9999999    4444444           
000000    00000000    0000000000000   00000000000000     00000      0000000   0000000    89999998         7  
000    00000000     0000000000000   000000000000000    00000       000000   00000000    000000   9999999     




Explore Full Features with Team or Enterprise Plans 
BOOK A DEMO 🚀
Enterprise-first experience
Flexible deployment options
Enhanced security support