Open Source AI
"Open source AI" gets used loosely, and the imprecision matters. In traditional software, open source means the source code is publicly available — you can read it, modify it, and redistribute it. In AI, the equivalent would mean releasing the training code, the training data, the training methodology, and the final model weights. Almost no one does all of that.
What most people mean when they say "open source AI" is open weights — the trained model parameters are publicly released. You can download and run them. You can fine-tune them on your own data. You can build products with them. What you typically don't get is the full training pipeline, the data, or the internal evaluations.
That distinction matters less for most users than it sounds. Open weights give you enormous practical freedom compared to a closed API.
What you can actually do with them
Run models privately. If you're working with sensitive documents — patient records, legal files, confidential business data — sending that text to OpenAI or Anthropic's servers may not be acceptable. Running a local model means your data never leaves your machine.
Fine-tune on your own data. The publicly available models are general-purpose. Fine-tuning lets you take a base model and train it further on your specific domain — your company's tone, a specialized vocabulary, a particular task. This is how organizations build internal AI tools that actually fit their workflows.
Remove API dependencies. Building a product on top of a closed API means you're subject to that API's pricing, rate limits, terms of service changes, and potential discontinuation. An open weights model is infrastructure you control.
Run offline. Air-gapped environments, remote locations, regulated industries that prohibit cloud data transmission — all become viable.
High-volume workloads. Once you've acquired the hardware, inference costs approach zero. At scale, this changes the economics dramatically compared to per-token API pricing.
Where they fall short: for casual users who just want to chat or write, cloud models are easier to start with and the best closed models are still more capable. The capability gap is real — though it's been closing faster than most expected.
What hardware do you actually need
This is the part most articles skip or vague out on. The short answer: it depends on what size model you want to run, and whether you have an NVIDIA GPU or Apple Silicon.
| Model size | Memory needed | Consumer option |
|---|---|---|
| 7B–13B | 8–10 GB VRAM | RTX 3080 / 4070, Mac 16 GB unified |
| 30B–70B | 20–48 GB VRAM | RTX 4090 (24 GB), Mac Studio / Pro 64–96 GB |
| 70B quantized | 16–24 GB VRAM | RTX 4090 with quality tradeoff |
| 405B+ (e.g. Llama 3.1) | 200 GB+ | Multi-GPU rig or Mac Pro 192 GB |
Quantization is a compression technique that reduces a model's numerical precision — from 16-bit floats to 8-bit or 4-bit integers. A quantized 70B model behaves almost as well as the full-precision version while fitting in roughly half the memory. Most local inference tools apply quantization automatically.
Apple Silicon deserves special mention. Mac chips use unified memory — the CPU and GPU share the same memory pool rather than having separate allocations. This means an M3 Max Mac Studio with 96 GB of unified memory can run 70B models in a way that a single consumer PC GPU with 24 GB VRAM simply can't. If you're considering local inference without a dedicated workstation GPU, high-memory Apple Silicon is surprisingly competitive.
If you want to try local models without buying new hardware first, start with a 7B or 13B model on whatever you already have. A modern laptop with 16 GB of RAM can run quantized 7B models — slowly, but functionally. The results will tell you whether it's worth investing in better hardware.
Getting started
Three tools make this accessible without much configuration:
Ollama is the easiest entry point. One command downloads and runs a model: ollama run llama3. It handles quantization, memory management, and serving automatically. This is where most people should start.
LM Studio provides a graphical interface for downloading and running models, with a chat UI built in. Good for non-developers who want to experiment without a terminal.
llama.cpp is the underlying inference engine that most of these tools are built on. If you want maximum control or are integrating local models into your own code, this is the foundation.
Where this is going
Open weights models have been improving faster than most observers predicted. Llama 3.1 405B — Meta's largest release — performs comparably to GPT-4 on a range of benchmarks. Models at the 70B scale routinely beat what GPT-3.5 could do, on consumer hardware that anyone can buy.
The direction is clear: the capability available at open weights will continue to close the gap with frontier closed models, with roughly a 12–18 month lag. Consumer hardware capable of running those models keeps improving. The ecosystem of fine-tuned variants — specialized for medical documentation, legal research, code generation, and hundreds of other domains — is growing rapidly.
The less settled question is safety. Releasing model weights is irreversible: once a capable model is public, it can't be un-released, and its downstream uses can't be controlled. Some researchers argue this is unacceptably dangerous as models become more capable. Others argue that open access enables auditing, transparency, and community-driven safety improvements that closed APIs don't allow. This debate doesn't have a clean resolution yet — and it connects directly to the questions the Perspective section addresses about who bears responsibility when AI is misused.
The AI Safety module in the Perspective section covers the open vs. closed debate in more depth, including the argument that open-sourcing powerful models is itself a safety risk and the counter-argument that closed models concentrate dangerous power in a small number of corporate hands.