Install Ollama Inside Docker Container on Linux

Ollama is an open-source tool that allows you to run large language models (LLMs) like Llama, Gemma, and others locally on the computer without needing cloud access. It features a simple command line interface and a REST API, making it easy to download, run, and manage models.

This tutorial explains how to install Ollama inside a Docker container on the Linux. Commands have been tested on Ubuntu.

Prepare environment

Make sure you have installed Docker in your system. If you are using Ubuntu, installation instructions can be found in the post.

Install Ollama

• Host network

Run the following command to create a container for Ollama that uses host network:

docker run -d --name=ollama --restart=always --network=host \
    -v /opt/ollama:/root/.ollama \
    ollama/ollama

To enable NVIDIA GPU support for faster inference, use the following command:

docker run -d --name=ollama --restart=always --network=host \
    -v /opt/ollama:/root/.ollama \
    --gpus all \
    ollama/ollama

Note: GPU acceleration requires the NVIDIA Container Toolkit to be properly installed on the machine. Without it, the --gpus all option will not work. Installation instructions for Ubuntu can be found in the post.

• User-defined bridge network

User-defined bridge network can be used for listening on different port. By default, Ollama service is listening on port 11434. It can be changed with -p option.

docker network create app-net

docker run -d --name=ollama --restart=always --network=app-net \
    -p 8080:11434 \
    -v /opt/ollama:/root/.ollama \
    ollama/ollama

To run Ollama with NVIDIA GPU support, use:

docker run -d --name=ollama --restart=always --network=app-net \
    -p 8080:11434 \
    -v /opt/ollama:/root/.ollama \
    --gpus all \
    ollama/ollama

Testing Ollama

Once the Ollama container is running, you can test it by pulling a model and generating a response.

First, pull a supported model (e.g., Llama 3 1B variant) using the following command:

docker exec -it ollama ollama pull llama3.2:1b

After the model is downloaded, you can test it with a simple API call using curl command:

curl http://192.168.0.227:11434/api/generate -d '{"model":"llama3.2:1b","stream":false,"prompt":"What is LLM?"}'

Note: Replace the IP address with the actual IP address of the system.

Uninstall Ollama

To completely remove Ollama, remove its container:

docker rm --force ollama

Remove Ollama image:

docker rmi ollama/ollama

Delete the Ollama data directory (downloaded models and settings):

sudo rm -rf /opt/ollama

If a user-defined bridge network was created, you can delete it as follows:

docker network rm app-net