How to Run AI Models Locally on Your Computer (and Your Phone): The 2026 Guide
By Danar Mustafa · The AI & Tech Society · Published November 17, 2026 · Last updated November 17, 2026
Short answer: To run an AI model locally, install a free app such as LM Studio or Ollama, check how much VRAM (or unified memory on a Mac) your machine has, and download an open-weight model that fits in it. With 8 GB you can run Gemma 4 E4B or Qwen3 8B. With 16 GB, Gemma 4 12B or gpt-oss-20b. With 24 GB, Qwen3.8 27B, one of the strongest local all-rounders right now. On a phone, Google’s AI Edge Gallery app runs Gemma 4 E2B offline in under 1.5 GB of memory. Once a model is loaded you can chat with it, or expose it as an OpenAI-compatible API at localhost and plug it into VS Code, Claude Code or your own scripts. No per-token bills, and your data never leaves the device.

Run AI on your own computer and on your phone: a step-by-step local AI guide for November 2026 Local AI in four numbers. Sources: Google, Unsloth, SentinelOne and Censys (January 2026).
This guide started with a video I found unusually useful: Tech With Tim’s “Local AI Explained: How to Run AI Models on Your Computer”. It walks through LM Studio click by click. I’ve kept his practical flow, then added what has changed since and what I think most beginner guides leave out: which models are worth downloading in November 2026, how context length silently eats your memory, how to run models on a phone, and how not to end up as one of the 175,000 AI servers sitting open on the internet.
Why would you run AI locally in 2026?
There are four good reasons, and one honest limitation.
Privacy comes first. A local model processes your prompt on your own hardware. Contracts, customer data, medical notes or unreleased code never travel to a third-party server. For many companies in the EU that alone settles the question.
Cost is the second. Once the model is downloaded, every token is free. If you run agents that loop for hours, or batch-process thousands of documents, the cloud bill adds up quickly. Locally you pay for electricity and hardware you may already own.
Third, it works offline: on a plane, in a basement lab, in a secure environment with no internet.
Fourth, control. You pick the exact model version, nobody deprecates it under your feet, and you can tune it, quantize it or swap it for another in minutes.
The limitation: the best local models are still behind the best cloud models on the hardest tasks, such as long multi-step agentic coding or deep research. A 27B model on a gaming GPU is impressive. It is not a frontier model running on a data center. The sensible setup for most people is hybrid: local for private, repetitive and offline work, cloud for the hardest problems.
What hardware do you need to run a local LLM?
The single number that matters most is fast memory: VRAM on a graphics card, or unified memory on an Apple Silicon Mac. The whole model, plus its working memory for the conversation, needs to fit there. If it doesn’t, the overflow spills into ordinary system RAM and speed collapses.
As Tim puts it in the video, the first thing to do after installing LM Studio is to check your VRAM, because “that’s going to determine what models you’re able to run.” In LM Studio you find it under Settings > Hardware.
A few rules of thumb:
- On Windows and Linux, the GPU’s VRAM is the limit. An RTX 3060 has 12 GB, a 4070 Ti Super 16 GB, a 3090 or 4090 24 GB and the RTX 5090 32 GB.
- On a Mac, the GPU shares memory with the system, so a 32 GB MacBook can use most of those 32 GB for a model. That is why Macs punch above their weight for local AI.
- No GPU at all? Small models (Phi-4 Mini, Gemma 4 E2B, Qwen3 4B) run on a CPU. Slowly, but they run.
- Keep 1 to 2 GB of headroom for the operating system and other apps.

Table showing which local AI models fit phones, 8 GB, 12 to 16 GB, 24 GB, 32 to 64 GB and 128 GB+ machines Pick the model by memory, not by hype. Suggested open-weight models as of November 2026.
If you’re thinking of buying hardware specifically for local AI, the 2026 options have widened. NVIDIA’s DGX Spark and AMD’s Strix Halo machines (Ryzen AI Max+ 395) both ship with 128 GB of unified memory, which fits models no single consumer GPU can. Apple’s Mac Studio with M5 Ultra has been shipping since September 22. The trade-off is memory bandwidth: Strix Halo’s 256 GB/s is far slower than a high-end GPU, so a big model fits but generates text more slowly. Prices are also elevated this year because of the ongoing DRAM shortage, so check current prices before you plan around a figure you read in a spring review.
Which local AI model should you download?
Start with the newest well-supported model that fits your memory with room to spare. Here’s my shortlist for November 2026:
| Your memory | Best all-rounder | Also worth trying | Notes |
|---|---|---|---|
| Phone | Gemma 4 E2B | Gemma 4 E4B (8 GB+ phones) | Via Google AI Edge Gallery |
| 8 GB | Gemma 4 E4B | Qwen3 8B (Q4) | Good for chat, summaries, simple code |
| 12–16 GB | Gemma 4 12B | gpt-oss-20b, Devstral Small 2 | Devstral is coding-focused |
| 24 GB | Qwen3.8 27B (Q4) | Gemma 4 26B A4B, Muse Glimmer 30B | The sweet spot right now |
| 32–64 GB | Qwen3.8 27B at Q6–Q8 | Gemma 4 31B | Same models, higher quality, longer context |
| 128 GB+ | gpt-oss-120b | Large MoE models | DGX Spark, Strix Halo, Mac Studio |
A few words on the names you’ll see.
Gemma 4 is Google’s open-weight family, released April 2, 2026 under the Apache 2.0 license, which means you can use it commercially without special terms. It comes in E2B and E4B (built for phones and laptops), 12B (added in June, runs on 16 GB and understands audio natively), a 26B A4B mixture-of-experts model that only activates about 3.8B parameters per token and therefore runs fast, and a 31B dense model. Google reports over 150 million downloads of the family. That’s a vendor figure, but the model is clearly one of the most-used open families this year.
Qwen3.8 27B from Alibaba is the model I’d pick for a 24 GB GPU or a 32 GB Mac. It’s Apache 2.0, has a 256K-token context window, and the 4-bit version from Unsloth (UD-Q4_K_XL) is 16.35 GiB. Community benchmarks put it at roughly 40 tokens per second on an RTX 3090 and 46 on a 4090. That is faster than you can read.
gpt-oss-20b is OpenAI’s open-weight model for the 16 GB class, and Muse Glimmer 30B is Meta’s Apache 2.0 model that many people prefer for writing. I covered Muse in my article on Meta’s Muse models.
Benchmarks for local models move monthly and are mostly community-run. Treat any “best model” list, including mine, as a starting point. Download two candidates and test them on your own tasks.
What do Q4, Q8 and GGUF mean?
Models are released in 16-bit precision, which is too big for most machines. Quantization compresses the weights to fewer bits. Q8 is near-lossless and about half the size. Q4 is about a quarter of the size and, for models of 8B parameters and up, usually loses surprisingly little quality. Below Q4 the quality drops faster.
GGUF is the file format used by llama.cpp, LM Studio and Ollama. MLX is Apple’s format, and on a Mac it’s often faster. LM Studio shows both and marks the one that suits your hardware.
Tim’s advice in the video is practical: in LM Studio’s model search, take the version with the thumbs-up icon. It’s the recommended quantization for your machine, and Q4 is generally a good default.
How do you set up LM Studio, step by step?
LM Studio is the easiest way to start if you like a visual interface. It’s free and runs on Mac, Windows and Linux. The current version is in the 0.4.2x series.

Seven steps from installing LM Studio to connecting a local model to coding tools The seven-step setup, based on Tech With Tim’s walkthrough and LM Studio’s documentation.
- Install LM Studio from lmstudio.ai/download. Note that LM Studio now also offers Bionic, a separate agent app. If you just want to run models, you want LM Studio itself.
- Check your memory under Settings > Hardware. Note the VRAM or unified memory figure.
- Search for a model with the magnifying glass. Pick one from the table above that fits, take the recommended (thumbs-up) quantization and download it.
- Load it with the advanced settings open. Click “Show advanced settings” when loading, because this is where you control memory use:
- Context length: how much text the model can hold in one conversation. Higher means more memory. More on this below.
- K/V cache quantization: compresses the conversation memory to Q8 or Q4. Q8 halves it with very little quality loss.
- Speculative decoding (MTP): makes generation faster on supported models. Stable in LM Studio since version 0.4.14 (May 2026).
- Tick “Remember settings” so you don’t have to set this every time.
- Chat and test. The chat supports system prompts, structured JSON output, document upload (RAG), a JavaScript sandbox and MCP plugins. Try your real tasks before you commit to a model.
- Start the server. Open the Developer tab and turn on the local server. LM Studio exposes an OpenAI-compatible API at
http://localhost:1234/v1and shows a ready-madecurlexample. - Connect your tools. Any app that speaks the OpenAI API can now use your local model (see below).
You can load several models at once if they fit in memory, for example a small fast model for autocomplete and a bigger one for chat.
Why does context length matter so much for memory?
This is the part most beginner guides skip, and it’s the most common reason a local model suddenly gets painfully slow.
Every token in the conversation is stored in a KV cache so the model doesn’t have to re-read everything for each new word. For Qwen3.8 27B that cache costs about 64 KiB per token at full (f16) precision. That sounds small until you multiply it out: a full 262K-token context needs about 16 GB just for the cache, as much as the model itself.

Bar chart showing model weights plus KV cache memory for Qwen3.8 27B at different context lengths On a 24 GB card, 96K tokens of context fits at full precision. With a Q8 cache, 128K fits. The full 262K needs a Q4 cache or more memory.
The practical rules:
- Don’t max out the context slider by default. Set what you actually need. 32K is plenty for chat; coding agents benefit from 64K to 128K.
- If you need long context, quantize the KV cache to Q8 first. It halves the cache with little quality loss. Q4 halves it again, with a bigger quality trade-off.
- If the first answer takes forever, memory has spilled out of VRAM. Tim’s fix: eject the model, lower the context length, and reload.
How do you make a local model faster?
Three settings make the biggest difference.
Speculative decoding with MTP. Some models, Qwen3.8 among them, can predict several tokens at once and then verify them. Both LM Studio and Ollama support it now. On Apple’s M5 Max with 128 GB, community tests of Qwen3.8 27B went from about 31 to 65–69 tokens per second with MLX plus MTP. On Strix Halo, from about 11 to 24–30. That’s roughly double, for free.
Use the right engine for your hardware. On a Mac, MLX versions are often faster than GGUF. On NVIDIA, make sure the full model is offloaded to the GPU (LM Studio shows a “GPU offload” slider; it should be at max).
Pick a mixture-of-experts model. Models like Gemma 4 26B A4B only activate a fraction of their parameters per token, so they generate much faster than a dense model of the same size. You still need memory for all the weights, though.
How do you use a local model from VS Code, Claude Code or Python?
Because LM Studio and Ollama both speak the OpenAI API, almost every AI tool can use them. You only need three things: the base URL, any API key (local servers don’t check it, but many tools require the field to be non-empty) and the model ID.
VS Code with Cline, as Tim shows in the video: choose “OpenAI Compatible” as the provider, paste http://localhost:1234/v1 as the base URL, type any API key and copy the model ID from LM Studio’s Developer tab.
Python, with the standard OpenAI library:
from openai import OpenAIclient = OpenAI(base_url="http://localhost:1234/v1", api_key="local")reply = client.chat.completions.create( model="qwen3.8-27b", # copy the exact ID from LM Studio messages=[{"role": "user", "content": "Summarize this contract clause in plain English: ..."}],)print(reply.choices[0].message.content)
Claude Code, Codex and other coding agents work too. The simplest route is Ollama, which has an Anthropic-compatible API and a launcher command:
ollama launch claude --model qwen3.8:27b
Be realistic here. Agentic coding is the hardest workload for local models, because the agent sends long contexts and many tool calls. A 27B model can handle well-defined tasks. For large refactors across a codebase, frontier cloud models are still clearly ahead. I wrote about how engineering leaders should think about these tool choices.
Should you use Ollama instead of LM Studio?
Use Ollama if you’re comfortable in a terminal or want a background service. Use LM Studio if you want to see and click everything. Many developers use both.
Ollama (version 0.40.x in November 2026) installs as a service, runs its API on port 11434, and gets you from zero to chatting in two commands:
ollama pull gemma4:12bollama run gemma4:12b
Other useful tags: gemma4:e2b, gemma4:e4b, qwen3.8:27b (about an 18 GB download) and gpt-oss:20b. Ollama also has an MLX engine on Macs and supports MTP. One thing to watch: Ollama now offers cloud models as well, so check that the model you’re running is actually local if privacy is the point.

Comparison of LM Studio, Ollama, llama.cpp, Google AI Edge Gallery and Bionic Five ways to run open models locally. Versions as of November 2026.
The other names in the table: llama.cpp is the engine underneath much of this ecosystem, for those who want maximum control and the newest features first. Bionic is LM Studio’s new agent app (version 1.2.1, released November 16) for agentic coding, research and document work. When LM Studio is running, Bionic uses it for local inference.
Can you run AI on your phone?
Yes, and it’s become surprisingly good. The easiest way is Google AI Edge Gallery, a free app on Google Play and the App Store (Android 12+ or iOS 17+). It downloads Gemma 4 models to the phone and runs them fully offline.
What you can do in it:
- chat with a model, including a “Thinking Mode” for harder questions
- ask questions about a photo (Ask Image)
- transcribe and summarize audio (Audio Scribe)
- try Agent Skills, small on-device agent tasks.
Start with Gemma 4 E2B, which runs in under 1.5 GB of memory. If you have a recent flagship with 8 GB or more of RAM, try E4B for better answers. Expect your phone to get warm and the battery to drain faster during long sessions. Some reviewers also report crashes on older or mid-range devices, so treat it as a capable demo and a private offline assistant, not yet a replacement for the cloud apps.
The bigger story is that on-device models are now good enough for translation, summarizing, and quick questions without a connection. Developers can build the same models into their own apps with Google’s LiteRT tooling.
How do you keep a local AI server secure?
This is the section I’d ask you not to skip.
When you turn on the local server, it listens for requests. By default, LM Studio and Ollama listen only on your own machine (localhost). The problem starts when people change that to share the model across a network, open a port on their router, or run it on a cloud server without authentication.
In January 2026, researchers at SentinelOne and Censys found 175,000 Ollama hosts exposed to the internet across 130 countries. Anyone could send them prompts, use their GPUs for free or pull information from them. Attackers noticed: a campaign researchers named Operation Bizarre Bazaar hijacked exposed AI servers and resold access (“LLMjacking”). In May, a critical Ollama vulnerability, CVE-2026-7482 (“Bleeding Llama”, CVSS 9.1), was disclosed and fixed in version 0.17.1.
My checklist:
- Never expose port 1234 or 11434 to the internet. If you need remote access, use a VPN such as Tailscale, or LM Studio’s LM Link feature, which is designed for connecting your own devices.
- Keep the apps updated. Bleeding Llama was patched quickly. Old installs stay vulnerable.
- Be careful with MCP plugins and agents. A local model with file system or shell access can do real damage if it follows a malicious instruction hidden in a document or web page. Give tools the narrowest permissions possible.
- Download models from known publishers (the official Google, Qwen, OpenAI, Meta or Unsloth pages) and check the license before commercial use. Gemma 4 and Qwen3.8 are Apache 2.0, but not every model is.
- Remember that “local” isn’t automatically “compliant”. If you process personal data, your GDPR obligations still apply. Local just makes them easier to meet.
What should you do if the local model is slow or crashes?
Most problems come down to memory.
| Symptom | Likely cause | Fix |
|---|---|---|
| First answer takes a minute or more | Model or context spilled out of VRAM | Eject, lower context length, reload |
| Speed drops as the chat gets longer | KV cache growing past your memory | Quantize the cache to Q8, or start a new chat |
| “Failed to load model” | Not enough memory | Pick a smaller model or a lower quantization (Q4) |
| Gibberish or repeated text | Wrong chat template or too low quantization | Use the recommended download; avoid below Q4 |
| Slow on a Mac | GGUF instead of MLX | Download the MLX version |
| Tool can’t connect | Server not running, or wrong model ID | Start the server in the Developer tab; copy the ID exactly |
| Phone app crashes | Model too large for the device | Switch from E4B to E2B |
FAQ
Is it free to run AI models locally? Yes. LM Studio, Ollama, llama.cpp and Google AI Edge Gallery are free, and open-weight models like Gemma 4 and Qwen3.8 are free to download. You pay only for hardware and electricity.
How much RAM do I need to run a local LLM? What matters is VRAM on a GPU or unified memory on a Mac. 8 GB runs small models like Gemma 4 E4B, 16 GB runs Gemma 4 12B or gpt-oss-20b, and 24 GB runs Qwen3.8 27B at 4-bit.
What is the best local AI model in November 2026? For a 24 GB GPU or 32 GB Mac, Qwen3.8 27B is one of the strongest all-rounders. For 12 to 16 GB, Gemma 4 12B. For phones, Gemma 4 E2B or E4B. Test two models on your own tasks before deciding.
Can I run AI on my phone without internet? Yes. Google AI Edge Gallery for Android and iOS downloads Gemma 4 models and runs them fully offline. Start with Gemma 4 E2B, which needs under 1.5 GB of memory.
Is LM Studio or Ollama better? LM Studio is easier for beginners because everything is visual. Ollama suits developers who prefer the command line or need a background service. Both offer OpenAI-compatible APIs and run the same models.
Can I use a local model with Claude Code or VS Code? Yes. Point the tool at the local server (http://localhost:1234/v1 for LM Studio, port 11434 for Ollama), enter any API key and the model ID. Ollama’s ollama launch command starts Claude Code with a local model directly.
Is local AI as good as ChatGPT or Claude? For chat, summaries, writing help and well-defined coding tasks, the best local models are good enough for daily use. For long agentic tasks and the hardest reasoning, frontier cloud models are still clearly ahead.
Is running a local AI server safe? It is safe as long as the server only listens on your own machine and you keep the software updated. Never expose ports 1234 or 11434 to the internet. In January 2026, researchers found 175,000 Ollama servers that were openly reachable.
Final take
A year ago, running AI locally was a hobby for people with expensive GPUs and patience. In November 2026 it’s a reasonable default for private and repetitive work. A 24 GB graphics card runs a model that would have been frontier-class not long ago, and a phone runs a capable assistant offline.
My advice: install LM Studio, check your memory, download one model from the table above and give it a real task from your week. Keep the context length modest, keep the server on localhost, and keep the cloud for the problems that need it. That hybrid setup is where most people and teams will end up, and the earlier you learn the local half, the better your choices about the cloud half will be.
Sources: Tech With Tim, “Local AI Explained: How to Run AI Models on Your Computer” (YouTube, 2026), the basis for the LM Studio walkthrough. LM Studio changelogs and documentation (versions 0.4.14–0.4.27; Bionic 1.2.0–1.2.1). Ollama release notes (v0.40.x). Google, Gemma 4 announcement and model card (April 2 and June 3, 2026) and the Google AI Edge Gallery app listings. Unsloth model files for Qwen3.8 27B. KV-cache and speed figures from OpenClawDC, PromptQuorum and DevToolLab local LLM guides (October–November 2026); these are community measurements and vary by setup. SentinelOne and Censys research on exposed Ollama hosts (January 2026). CVE-2026-7482 disclosure (May 5, 2026). Download counts are vendor figures. Model rankings are my own assessment based on these sources and change quickly.
Discover more from The Tech Society
Subscribe to get the latest posts sent to your email.