Running DeepSeek on Ollama: A Private, Local AI Setup That Works
Over the past few weeks, I’ve been experimenting with running DeepSeek on Ollama, and I’m genuinely impressed. The ability to run AI models locally—without sending data over the Internet—means my chats remain private and confidential. For anyone concerned with data security, this is a huge win.
Hardware and Setup
The hardware requirements aren’t outrageous. I had a 24GB NVIDIA GPU left over from a previous project. I had bought it on Facebook Marketplace about 18 months ago, and though it had been sitting unused for over a year, it’s still a solid piece of hardware.
Setting up Ubuntu was surprisingly straightforward. I chose not to virtualize the machine or its operating system, which made the setup process a lot smoother. Installing the necessary drivers to get the GPU working was easy. While virtualization is an option, GPU passthrough can be tricky, so I’ve saved that for a potential future project.
Comparing with Mac Studio
Before this, I ran LLaMA 3 on a 32GB M1 Mac Studio, which handled most open-source models without breaking a sweat. Naturally, I expected a Linux setup with more cores and a dedicated GPU to be significantly faster. But in practice, it wasn’t. The Mac Studio held its own and, in many ways, performed just as well.
Why Local AI Matters
I truly believe this is one of the simplest and most effective ways to bring private, secure AI capabilities to any organization. There’s a wide range of open-source large language models (LLMs) available, each with its own strengths. In reality, most people don’t need the full power of GPT-4 for everyday tasks.
Going the open-source route is not only free but also gives me more control—especially when fine-tuning models to fit my needs. Running LLMs on in-house hardware also lets multiple users share the GPU, which helps distribute the cost efficiently.
My Use Case: Code Review
I primarily use DeepSeek to check my code. Since I work on a lot of collaborative projects, it’s great to have an AI assistant that can review my work instantly—no need to wait for a colleague to be available. This has been a huge productivity boost.
I'm also planning to test other open-source LLMs to evaluate their strengths in areas like coding, writing, summarization, and more.
Cost Savings for Organizations
A typical subscription for a professional-grade AI service costs around $20 per month. For a small organization with 20 people, that adds up quickly—several hundred dollars each month, and a few thousand dollars per year.
Some of my clients, especially biotech companies, have proprietary IP they can’t risk uploading to a cloud-based AI provider. Others are consultants who don’t have the legal rights to send confidential data to third parties. For these use cases, setting up a private AI server is a perfect solution.
Final Thoughts
Local, private AI is no longer just a dream—it’s here and working, even on consumer hardware. With open-source LLMs evolving rapidly, I think we’ll see more and more organizations making the switch to in-house AI tools.
If you’re thinking about it, now’s a great time to dive in.
Hardware and Setup
The hardware requirements aren’t outrageous. I had a 24GB NVIDIA GPU left over from a previous project. I had bought it on Facebook Marketplace about 18 months ago, and though it had been sitting unused for over a year, it’s still a solid piece of hardware.
Setting up Ubuntu was surprisingly straightforward. I chose not to virtualize the machine or its operating system, which made the setup process a lot smoother. Installing the necessary drivers to get the GPU working was easy. While virtualization is an option, GPU passthrough can be tricky, so I’ve saved that for a potential future project.
Comparing with Mac Studio
Before this, I ran LLaMA 3 on a 32GB M1 Mac Studio, which handled most open-source models without breaking a sweat. Naturally, I expected a Linux setup with more cores and a dedicated GPU to be significantly faster. But in practice, it wasn’t. The Mac Studio held its own and, in many ways, performed just as well.
Why Local AI Matters
I truly believe this is one of the simplest and most effective ways to bring private, secure AI capabilities to any organization. There’s a wide range of open-source large language models (LLMs) available, each with its own strengths. In reality, most people don’t need the full power of GPT-4 for everyday tasks.
Going the open-source route is not only free but also gives me more control—especially when fine-tuning models to fit my needs. Running LLMs on in-house hardware also lets multiple users share the GPU, which helps distribute the cost efficiently.
My Use Case: Code Review
I primarily use DeepSeek to check my code. Since I work on a lot of collaborative projects, it’s great to have an AI assistant that can review my work instantly—no need to wait for a colleague to be available. This has been a huge productivity boost.
I'm also planning to test other open-source LLMs to evaluate their strengths in areas like coding, writing, summarization, and more.
Cost Savings for Organizations
A typical subscription for a professional-grade AI service costs around $20 per month. For a small organization with 20 people, that adds up quickly—several hundred dollars each month, and a few thousand dollars per year.
Some of my clients, especially biotech companies, have proprietary IP they can’t risk uploading to a cloud-based AI provider. Others are consultants who don’t have the legal rights to send confidential data to third parties. For these use cases, setting up a private AI server is a perfect solution.
Final Thoughts
Local, private AI is no longer just a dream—it’s here and working, even on consumer hardware. With open-source LLMs evolving rapidly, I think we’ll see more and more organizations making the switch to in-house AI tools.
If you’re thinking about it, now’s a great time to dive in.