Stop Paying for LLM APIs - Run Them Free on Google Colab

The simplest way to think about Ollama: it's a package manager for LLMs, with a built-in inference server.

Pull a model the same way you'd pull a Docker image:

That's it. Ollama downloads the model, stores it locally, and starts serving it through a REST API on localhost:11434. The API is OpenAI-compatible out of the box - and since v0.14.0, it also supports the Anthropic Messages API endpoint natively. That means any code written against the OpenAI or Anthropic Python SDKs works with Ollama by changing one line.

The model library covers most of what developers actually use:

• Meta Llama 3 (8B, 70B) - strong general reasoning

• Google Gemma 3 (4B, 8B, 27B) - fast, efficient, beginner-friendly

• Mistral 7B - reliable all-rounder with good instruction following

• Alibaba Qwen3 (8B, 14B) - strong multilingual and coding performance

• Microsoft Phi-3 (3.8B) - surprisingly capable at low VRAM

• DeepSeek-R1 (8B) - chain-of-thought reasoning built in

Over 100 models are available at ollama.com/search, most with multiple quantization levels. The :q4_0 suffix cuts VRAM usage by roughly 40% - useful when you're working with constrained hardware.

Normally, Ollama requires a local GPU. That's the catch for most developers. A decent GPU for running 8B models starts around $300–500 used, and driver setup on Linux can eat a full afternoon. That's where Colab changes things entirely.

Google Colab's free tier gives you access to an NVIDIA T4 GPU with 16 GB of VRAM. For context, that's enough to run Llama 3 8B, Gemma 3 8B, Mistral 7B, and Qwen3 8B - comfortably, with headroom to spare. You get this from a browser tab, in about 60 seconds of setup, at no cost.

There's no hardware to buy. No CUDA toolkit to install. No driver version conflicts. Colab handles all of that - the GPU is already configured when the runtime starts. For anyone who's spent time getting local GPU inference to work, that's not a small thing.

The practical case for Colab over local setup:

• Zero upfront cost - free T4 GPU vs. $300–500+ for a used GPU

• No CUDA configuration - runtime comes pre-configured with NVIDIA drivers

• Browser-based access - works on any machine, including laptops without discrete GPUs

• Fast iteration - spin up a fresh environment in under two minutes

• Ideal for experimentation - test new models without committing disk space locally

For students, indie developers, and anyone building LLM-powered tools on a budget, this combination removes the last real barrier to self-hosted inference.

None of this is magic. There are real constraints worth knowing before you build anything on top of this setup.

Session timeouts. Free Colab sessions run for up to 12 hours of active use, then disconnect. The runtime resets completely - everything in the VM's filesystem is wiped. If you haven't configured persistent storage, you lose your downloaded models and have to pull them again.

Model re-downloads. A 7B model is typically 4–6 GB. Pulling it fresh every session adds 3–5 minutes of dead time. This guide covers how to mount Google Drive and redirect Ollama's model storage path, which eliminates the problem entirely.

Ngrok URL rotation. The free Ngrok tier generates a new public URL each time you restart the tunnel. Anything that hardcodes the URL - a frontend app, a Claude Code config, a .env file - breaks when the session resets. Paid Ngrok plans support static domains; the free tier doesn't.

Not production-ready. Colab sessions aren't persistent, the URLs change, and there's no uptime guarantee. For production workloads, look at RunPod, Lambda Labs, or Vast.ai - GPU cloud providers where you rent dedicated compute by the hour and keep it running.

For development and prototyping, free Colab hits the sweet spot. For anything running in production with real users, you want dedicated compute - but that's a problem you solve after you've validated the idea.

Navigate to colab.research.google.com and sign in with your Google account. Click New Notebook in the top-left corner. You'll land in a blank notebook with a single empty code cell - that's your starting point.

This step is non-negotiable. Colab defaults to CPU, which Ollama will use silently if you don't change it.

1. In the top menu, click Runtime

2. Select Change Runtime Type

3. Under Hardware Accelerator, choose T4 GPU

4. Click Save

Colab will reconnect the runtime - this takes about 10–15 seconds. When it completes, you'll see a green checkmark and RAM/Disk indicators in the top-right corner confirming the GPU is active.

Stick with T4 GPU. Ollama is built around CUDA and runs natively on NVIDIA GPUs. The v5e-1 TPU option is a Google Cloud TPU - a different architecture entirely, optimized for TensorFlow/JAX workloads, not general LLM inference. Ollama doesn't benefit from TPU acceleration, and compatibility is unreliable. T4 is the right call here.

Pro tip: Colab sometimes shows "GPU not available" during high-demand periods, especially on free accounts. If that happens, try switching to a different runtime type and back, or disconnect and reconnect the runtime. Peak hours (US afternoon/evening) are the worst for availability - early morning often gets you a GPU faster.

Colab notebooks support two ways to run shell commands: the built-in terminal (accessible via the left sidebar terminal icon) or the ! prefix directly in a code cell. Either works. For this setup, code cells with ! are easier to re-run if something goes wrong.

In your first code cell, run:

Neither package is large. The install typically completes in under 30 seconds on a fresh Colab runtime. If apt-get update hangs for more than two minutes, disconnect and reconnect the runtime - it's a known Colab networking quirk that clears itself on reconnect.

One thing to watch for: If the cell output shows WARNING: apt does not have a stable CLI interface - ignore it. That's a standard Colab warning, not an error. Your packages installed correctly if the final line reads 0 upgraded, N newly installed.

With the runtime configured and dependencies in place, the environment is ready to receive Ollama.

In a new code cell, run:

What happens when you run this:

1. The script fetches the latest Ollama binary for your platform (Linux x86_64 on Colab)

2. It detects the available GPU - in this case, the NVIDIA T4 via Colab's pre-installed CUDA drivers

3. It installs Ollama to /usr/local/bin/ollama

4. It starts the Ollama server as a background process on port 11434

You'll see output like Ollama is running on http://localhost:11434 when it completes. That means the API server is live and ready to accept requests.

No manual CUDA configuration needed. Colab pre-installs NVIDIA drivers and the CUDA toolkit for any GPU runtime - Ollama picks them up automatically through the install script. This is one of the genuine advantages of running on Colab vs. a raw Linux server, where CUDA setup alone can take an hour.

Before pulling any models, confirm the server is actually listening on the right port:

In the output, look for this line:

That confirms Ollama is bound to all interfaces on port 11434 and ready to receive API calls.

You can also do a quick sanity check with curl:

Expected response: Ollama is running

If you get Connection refused instead, the server didn't start cleanly. The most common cause on Colab is a cold-start race condition - the script finishes but the process hasn't fully initialized.

Fix: run the below command.

Colab ships with Python 3 pre-installed, but it's worth a quick check before installing SDK packages in the next steps:

Expected output is Python 3.10.x or higher and pip 23.x or higher. If either command returns an error (unlikely on Colab, but possible after a runtime reset), use python3 and pip3 explicitly in all subsequent commands.

What you have at this point: A running Ollama inference server on a free T4 GPU, accessible via REST API at localhost:11434, with full OpenAI and Anthropic SDK compatibility waiting to be used. The next step is choosing and pulling the right model - and that decision depends on how much of the T4's 16 GB VRAM you want to use.

Ollama's full model library is at ollama.com/search. Over 100 models are available, filterable by task category: general chat, code generation, reasoning, vision, multilingual, and embedding. Most popular models offer multiple size variants - a 3b, 8b, and 27b version of the same architecture, for example - so you can match capability to hardware.

For Colab's free T4, focus on the 4B–8B parameter range. That's where the performance-to-VRAM ratio is strongest - capable enough for real development work, well within the 16 GB limit.

Every model in Ollama's library is available in multiple quantization levels. The default pull fetches a Q4 or Q4_K_M quantized version automatically - but you can be explicit:

Once you've picked a model, pull it:

The download takes 2–5 minutes depending on model size and Colab's network throughput. You'll see a progress bar with download speed and estimated time remaining.

Expected output shows the model name, size, and last modified timestamp:

If the list is empty after pulling, the download likely failed silently - common with Colab's network throttling on large files. Re-run the pull command; Ollama resumes from where it left off.

Colab's VM filesystem is ephemeral by design. Google spins up a fresh containerized environment each time you connect - nothing carries over from previous sessions except what's stored outside the VM. Google Drive, by contrast, persists indefinitely. It's mounted as a filesystem path inside Colab, which means any directory on Drive looks and behaves like a local folder from Ollama's perspective.

The free 15 GB Google Drive tier is enough for two or three mid-size models simultaneously:

If Drive is running low, the :q4_0 quantized variants of each model save roughly 1–2 GB per model compared to defaults.

In a new code cell, run:

Colab will open an authentication dialog asking you to sign in to your Google account and grant Drive access. This authentication persists for the browser session - you won't be prompted again unless you clear cookies or use a different browser.

After mounting, your Drive is accessible at /content/drive/MyDrive/.

Ollama respects a single environment variable - OLLAMA_MODELS - to control where models are stored. Setting it before starting the server is all that's needed:

The & at the end runs ollama serve in the background so it doesn't block the cell. Wait 3–5 seconds after running this before executing the next cell - the server needs a moment to initialize before it's ready to accept requests.

Critical order: OLLAMA_MODELS must be exported before ollama serve starts. If you set the variable after the server is already running, it won't pick it up - you'll need to pkill ollama and restart.

Pull a small model to confirm storage is pointing to Drive:

Then check the Drive directory:

You should see a manifests and blobs subdirectory - Ollama's internal model storage structure. If the directory is empty after pulling, the OLLAMA_MODELS variable wasn't set correctly. Run echo $OLLAMA_MODELS to confirm it's pointing to the Drive path.

The real payoff comes from combining everything into a single startup cell that runs at the top of every session:

On the first session, run ollama pull <model> after this cell to download your model to Drive. From the second session onward, skip the pull entirely - the model is already there. The server finds it in Drive and starts serving it within seconds of startup.

One server, three clients. Pick the one that matches your existing codebase.

The Ollama SDK is the most direct way to interact with the server. It auto-detects localhost:11434 - no URL configuration needed.

The ChatResponse object gives you direct access to .message.content for the text output, plus metadata like token counts and model name. The SDK also supports streaming responses out of the box - pass stream=True to chat() and iterate over the generator to print tokens as they arrive, rather than waiting for the full response.

Best for: Projects built specifically around Ollama, use cases that need streaming, or workflows that call Ollama-specific features like ollama.embeddings() for vector generation.

This is the most immediately useful option for most developers. If you've written code against openai.ChatCompletion or openai.responses.create, you're two lines away from running it against a free local model instead.

Two configuration lines swap the entire backend from OpenAI's servers to your Colab GPU. The rest of your code stays identical.

The trailing slash on base_url matters. http://localhost:11434/v1/ works. http://localhost:11434/v1 without the slash returns a 404 on most OpenAI SDK versions. It's the single most common mistake when migrating existing code to Ollama.

This compatibility extends to the entire OpenAI SDK ecosystem:

• LangChain - use ChatOpenAI(base_url='http://localhost:11434/v1/', api_key='ollama')

• LlamaIndex - works with any OpenAI-based LLM class

• Continue (VS Code extension) - configure the Ollama provider in config.json

• Any tool that accepts a custom base_url - works without modification

Best for: Migrating existing OpenAI-powered apps to local inference, integrating with LangChain or LlamaIndex, or any tool in the OpenAI SDK ecosystem.

This is the option most tutorials skip entirely, and it's genuinely useful for a growing slice of the developer community.

Since Ollama v0.14.0, the server natively supports the Anthropic Messages API at /v1/messages. That means the anthropic Python SDK connects to Ollama directly - no adapter, no wrapper, no middleware.

Note that base_url here does not include /v1/ - the Anthropic SDK appends that path internally. Using http://localhost:11434 is correct. Using http://localhost:11434/v1/ will double the path prefix and break the connection.

The Claude Code integration. This is where the Anthropic SDK option becomes particularly powerful. Claude Code respects the ANTHROPIC_BASE_URL environment variable. Set it to your Ngrok tunnel URL (covered in the next step) and Claude Code routes all its inference requests to your Colab-hosted Ollama server - running open-source models through the exact same interface you'd use with Claude:

From that point, Claude Code calls your Gemma3 or Llama3 model on Colab's T4 GPU instead of Anthropic's API. No API costs. No rate limits.

Best for: Developers working with Claude Code, existing anthropic SDK pipelines, or anyone building against the Anthropic Messages API who wants a free local alternative for development.

If you haven't already, sign up at dashboard.ngrok.com/signup. The free tier supports one active tunnel - exactly what this setup needs.

After verifying your email, navigate to Your Authtoken in the left sidebar and copy the token. It looks like a long alphanumeric string. Keep it accessible - you'll paste it into Colab in the next step.

Ngrok isn't pre-installed on Colab runtimes, but it's in an official apt repository. Run this in a code cell:

This installs the official Ngrok agent - not a community package, not a Python wrapper. The agent binary is what creates and manages the tunnel directly. Installation takes about 20–30 seconds.

Replace $YOUR_AUTHTOKEN with the token you copied from the dashboard:

Then start the tunnel:

The --host-header flag is critical. Without it, Ngrok forwards requests with the public hostname in the Host header - and Ollama rejects those requests because it only accepts localhost as a valid host. This flag rewrites the header to localhost:11434 before the request hits Ollama, so the server accepts it cleanly.

You'll see output like this:

That public URL is your Ollama server. Copy it.

With the tunnel active, your Colab-hosted Ollama server is reachable from anywhere. Swap localhost:11434 for the Ngrok URL in any of the SDK configurations from the previous step:

The same URL also works with LangChain's ChatOpenAI, the Continue VS Code extension, LlamaIndex, and any other tool that accepts a custom base URL. Your Colab GPU becomes a remote LLM endpoint available to your entire local development environment.

Free tier limitation: The Ngrok URL changes every time you restart the tunnel - it's not a static domain on the free plan. Anything that hardcodes the URL (a .env file, a config file, a mobile app) breaks on session restart. If URL stability matters for your workflow, Ngrok's paid plans start at around $8/month and include a fixed subdomain.

Create a file called ollama.yaml in the Colab environment:

Then start the tunnel with the policy attached:

Every request now requires an Authorization: Basic header with your credentials. Requests without it receive a 401 Unauthorized response before they ever reach Ollama.

To call the secured endpoint from an SDK, encode your credentials and pass the header:

If you only ever call the endpoint from a fixed IP - your home network, your office - this is the cleanest option. In the Ngrok dashboard, go to Cloud Edge → IP Restrictions and add your IP address. All requests from any other IP return 403 Forbidden before the tunnel even forwards them.

Verify auth is working:

Symptom: The install script runs but curl http://localhost:11434 returns Connection refused.

Fix: Run the below command. First-run failures on cold Colab runtimes are common:

If the second run also fails, check the install output for CUDA detection errors. A line like no NVIDIA GPU detected means the runtime didn't connect to the T4 - disconnect the runtime, reconfirm T4 GPU is selected under Runtime → Change Runtime Type, and reconnect.

Symptom: sudo ss -tuln doesn't show 0.0.0.0:11434.

Fix: The Ollama process isn't running. Start it manually:

Wait 3–5 seconds, then re-check with sudo ss -tuln. This also applies after any runtime restart - Ollama doesn't auto-start between sessions.

Symptom: ollama pull hangs indefinitely, errors out, or downloads at under 1 MB/s.

Fix: Colab's network throughput varies significantly by time of day. Two approaches:

1. Try a smaller model first (ollama pull phi3 - 2.2 GB) to verify the pipeline works before committing to a 6 GB download

2. If a pull stalls partway through, re-run the same command - Ollama resumes from the last completed chunk

OOM crash during inference: If the model loads but the server crashes when you send a request, you've hit the T4's 16 GB VRAM ceiling. Check the VRAM table in Step 3 and switch to a smaller model or a :q4_0 quantized variant.

Symptom: ollama list returns empty even though models were previously pulled to Drive.

Diagnose in order:

The most common cause: ollama serve was started before OLLAMA_MODELS was exported. The server caches the storage path at startup - setting the variable afterward has no effect.

Fix: pkill ollama, re-export the variable, then restart with ollama serve &.

Symptom: The Ngrok URL returns a 502 Bad Gateway or stops responding.

Fix: The tunnel process was killed when Colab's idle timeout hit, or the free session limit (2-hour tunnel for unverified accounts) expired. Re-run the tunnel command to get a new URL:

For automated tunnel management without manually re-running the command, use the pyngrok Python library directly inside the notebook:

pyngrok keeps the tunnel alive as long as the notebook kernel is running - no separate terminal process needed.

Symptom: client.responses.create() or client.chat.completions.create() raises a 404 Not Found error.

Two causes, check in order:

1. Missing trailing slash - base_url must be http://localhost:11434/v1/ with the trailing slash. http://localhost:11434/v1 without it routes to the wrong endpoint on most OpenAI SDK versions.

2. Wrong model name - the model name in the API call must exactly match what Ollama has pulled. Run ollama list to see available models and copy the name character-for-character:

Use gemma3 or gemma3:latest - not gemma-3 or gemma3b.

Symptom: client.messages.create() raises a Connection refused or APIConnectionError.

Two causes:

1. Ollama version too old - Anthropic Messages API support requires v0.14.0 or higher. Check:

If it returns a version below 0.14.0, re-run the install script to get the latest release.

2. Wrong base_url format - The Anthropic SDK appends /v1/messages internally. Your base_url should be http://localhost:11434 - without any path suffix. Using http://localhost:11434/v1 doubles the path and breaks routing.