9 Best Local/Offline LLMs You Can Try Right Now

With quantum LLMs now available on HuggingFace and AI ecosystems like H20, Text Gen, and GPT4All allowing you to load LLM weights on your computer, you now have an option for free, flexible, and secure AI. Here are the 9 best local/offline LLMs you can try right now!

1. Hermes 2 Pro GPTQ

9 Best Local/Offline LLMs You Can Try Right Now Picture 1

Hermes 2 Pro is a state-of-the-art language model fine-tuned by Nous Research. It uses an updated and compact version of the OpenHermes 2.5 dataset, along with the newly introduced Function Calling and JSON datasets developed by the company. The model is based on the Mistral 7B architecture and has been trained on 1,000,000 instructions/conversations of GPT-4 quality or better, mostly synthetic data.

2. Zephyr 7B Beta

Zephyr is a series of language models trained to act as helpful assistants. Zephyr-7B-Beta is the second model in the series, fine-tuned from Mistral-7B-v0.1 using Direct Preference Optimization (DPO) on a mix of publicly available synthetic datasets.

3. Falcon Instruct GPTQ

9 Best Local/Offline LLMs You Can Try Right Now Picture 2

This quantized version of Falcon is based on a decoder-only architecture fine-tuned on TII's raw Falcon-7b model. The base Falcon model is trained using 1.5 trillion outstanding tokens sourced from the public Internet. As an Apache 2-licensed, command-based decoder-only model, Falcon Instruct is perfect for small businesses looking for a model to use for language translation and data ingestion.

4. GPT4ALL-J Groovy

GPT4All-J Groovy is a decoder-only model tuned by Nomic AI and licensed under Apache 2.0. GPT4ALL-J Groovy is based on the original GPT-J model, which is known to be great at generating text from prompts. GPT4ALL-J Groovy has been tuned into a conversational model, which is great for fast and creative text generation applications. This makes GPT4All-J Groovy ideal for content creators in assisting them with their writing and composition, whether it is poetry, music, or stories.

5. DeepSeek Coder V2 Instruct

9 Best Local/Offline LLMs You Can Try Right Now Picture 3

DeepSeek Coder V2 is an advanced language model that enhances programming and mathematical reasoning. DeepSeek Coder V2 supports multiple programming languages ​​and provides extended context length, making it a versatile tool for developers.

6. Mixtral-8x7B

9 Best Local/Offline LLMs You Can Try Right Now Picture 4

Mixtral-8x7B is a mixture of expert (MoE) model developed by Mistral AI. It has 8 experts per MLP, totaling 45 billion parameters. However, only two experts are activated per token during inference, making it computationally efficient, with speed and cost comparable to a 12 billion parameter model.

7. Wizard Vicuna Uncensored-GPTQ

Wizard-Vicuna GPTQ is the quantum version of Wizard Vicuna based on the LlaMA model. Unlike most LLMs released to the public, Wizard-Vicuna is an uncensored model with de-linking. This means that the model does not have the same safety and ethical standards as most other models.

8. Orca Mini-GPTQ

9 Best Local/Offline LLMs You Can Try Right Now Picture 5

Looking to test a model trained using a unique learning approach? Orca Mini is an informal implementation of Microsoft's Orca research papers. The model is trained using a teacher-student learning approach, where the dataset is filled with explanations rather than just prompts and feedback. This should theoretically make the student smarter, as the model can understand the problem rather than just look for input and output pairs as a typical LLM would.

9. Llama 2 13B Chat GPTQ

9 Best Local/Offline LLMs You Can Try Right Now Picture 6

Llama 2 is the successor to the original Llama LLM, offering improved performance and flexibility. The 13B Chat GPTQ variant is tuned for conversational AI applications optimized for English dialogue.

Some of the models listed above come in multiple spec versions. Generally, higher spec versions will produce better results but require more powerful hardware, while lower spec versions will produce lower quality results but can run on lower-end hardware. If you're not sure whether your PC can run a model, try the lower spec version first, then move on until you feel the performance drop is no longer acceptable.

Lesley Montoya

Update 17 December 2024