Self Hosted ChatGPT using Ollama

Setting up Ollama AI is a great way to dive into the world of artificial intelligence without the hassle. Whether you want help with content creation, answering complex questions, or just automating some of your routine tasks, Ollama makes AI accessible and pretty intuitive. It’s built for everyone from curious beginners to tech enthusiasts who want to get hands-on with AI in a way that’s practical and fun. If you’re ready to see what AI can really do for you, Ollama is a solid place to start!

Ollama runs best when paired with a GPU however you don't necessarily need one, it just speeds things up considerably. I've tested with an RTX 3070 and 4060 but you can go as far back as a GTX 1050. You can checkout Nvidia's CUDA GPU list here. It just needs to have compute capability of at least 5.0.

During my testing I used Debian 12. You can checkout instructions for installing Nvidia drivers here. Once installed, ensure you have disabled Secure Boot in the BIOS as this causes the incorrect kernel to load.

Installing Ollama is as simple as running this command...

curl -fsSL https://ollama.com/install.sh | sh

Once installed, you can choose from a number of different Ollama models that are available here.

I used llama3.2 - this utilises around 4GB of graphics memory when in use.

To install this, run this command...

ollama pull llama3.2

Then to run the model use this command...

ollama run llama3.2

You can run this command each time you want to use the model.

I'm quite happy with the text prompt on the command line however you may wish to add a web gui for the authentic ChatGPT feel. This is completely optional. I recommend using this project as it's regularly kept up to date.