The best used GPU for running local LLMs
Look at VRAM first and memory bandwidth second. Everything else on the spec sheet matters less than people think.
A model quantised to 4 bits needs a bit over half a gigabyte per billion parameters, plus room for the context. So an 8B model wants about 6 GB, a 14B model about 10, a 32B model about 20, and a 70B model needs 40 or more, which means two 24 GB cards. Once it fits, tokens come out roughly in proportion to memory bandwidth. A 3090 at 936 GB/s is close to three times quicker than a P40 at 346 on the same model.
| Card | Memory | Bandwidth | 8B | 14B | 32B | 70B | Idle | Power/yr | Guide price |
|---|---|---|---|---|---|---|---|---|---|
| Tesla P4 | 8 GB | 192 GB/s | 7 W | $22 | $90–150 | ||||
| RTX 3080 | 10 GB | 760 GB/s | 15 W | $72 | $450–550 | ||||
| RTX 2080 Ti | 11 GB | 616 GB/s | 15 W | $61 | $300–350 | ||||
| RTX 3060 12GB | 12 GB | 360 GB/s | 13 W | $45 | $340–400 | ||||
| RTX A2000 12GB | 12 GB | 288 GB/s | 6 W | $20 | $450–600 | ||||
| Tesla P100 | 16 GB | 732 GB/s | 28 W | $79 | $180–260 | ||||
| RTX 4060 Ti 16GB | 16 GB | 288 GB/s | 7 W | $36 | $650–750 | ||||
| Tesla T4 | 16 GB | 320 GB/s | 10 W | $25 | $700–1,000 | ||||
| RTX A4000 | 16 GB | 448 GB/s | 10 W | $36 | $850–1,100 | ||||
| Tesla P40 | 24 GB | 346 GB/s | 10 W | $54 | $350–450 | ||||
| RX 7900 XTX | 24 GB | 960 GB/s | 15 W | $77 | $1,120–1,310 | ||||
| RTX 3090 | 24 GB | 936 GB/s | 25 W | $91 | $1,500–1,700 | ||||
| RTX 4090 | 24 GB | 1008 GB/s | 20 W | $99 | $3,250–3,550 | ||||
| MI50 32GB | 32 GB | 1024 GB/s | 20 W | $76 | $630–900 |
Full circle: the model fits on one card at 4-bit. Half: it needs two. Power a year is at $0.18/kWh, on all day and mostly idle. Guide prices are eBay US asking prices checked 15 November 2026.
What to buy
For a first card, the RTX 3060 12GB. It holds a 14B model, it runs CUDA, so everything supports it, and it idles at about 13 W. Make sure the listing says 12GB. There's an 8 GB version with a narrower bus and it's a worse card for this.
If you need 24 GB and don't have $1,500, the Tesla P40 is still the cheap way in, at $350 to $450. Know what you're getting. It's passively cooled and needs a fan strapped to it outside a server. It takes an 8-pin CPU lead, not a PCIe one. And it's Pascal: Nvidia's 580 drivers are its last and CUDA 13 has already dropped it. llama.cpp still runs it well. Every update it misses makes it a bit more work to keep going.
The RTX 3090 is what most people end up with. It has 24 GB of fast memory and it runs CUDA. In late 2026, used ones are asking more than the $1,499 the card launched at. The RX 7900 XTX has the same 24 GB and nearly the same bandwidth for a few hundred less, and llama.cpp and Ollama are fine on ROCm or Vulkan now. If you don't need CUDA for anything else, it's the better buy.
The MI50 32GB was the bargain of 2025. Asking prices have more than doubled since the summer and ROCm has dropped it. It's also passive like the P40. Not at $700.
Skip the 3080 for this. 10 GB is too small for 14B with any context, and it pulls 320 W.
Idle power is the bill you forget
A box that runs Ollama sits idle most of the day. A 3090 idles around 25 W, which is about $40 a year at 18 cents before you've asked it anything. Two of them in an old dual-Xeon workstation idling at 90 W is a $200-a-year habit. If the machine only needs to be on when you're using it, turn it off. If it has to be on all the time, the idle figure in the table matters more than the load figure.
Tick Local AI in the planner to price a card with a host machine, or tick "The card goes in a PC I have" to rank cards on their own.
Guide prices checked 15 November 2026. Watts are typical readings at the wall; yours will vary.