labdeals

The best used GPU for running local LLMs

Look at VRAM first and memory bandwidth second. Everything else on the spec sheet matters less than people think.

A model quantised to 4 bits needs a bit over half a gigabyte per billion parameters, plus room for the context. So an 8B model wants about 6 GB, a 14B model about 10, a 32B model about 20, and a 70B model needs 40 or more, which means two 24 GB cards. Once it fits, tokens come out roughly in proportion to memory bandwidth. A 3090 at 936 GB/s is close to three times quicker than a P40 at 346 on the same model.

CardMemoryBandwidth8B14B32B70BIdlePower/yrGuide price
Tesla P48 GB192 GB/sFits on one cardNeeds two cardsDoesn't fitDoesn't fit7 W$22$90–150
RTX 308010 GB760 GB/sFits on one cardNeeds two cardsDoesn't fitDoesn't fit15 W$72$450–550
RTX 2080 Ti11 GB616 GB/sFits on one cardNeeds two cardsDoesn't fitDoesn't fit15 W$61$300–350
RTX 3060 12GB12 GB360 GB/sFits on one cardFits on one cardNeeds two cardsDoesn't fit13 W$45$340–400
RTX A2000 12GB12 GB288 GB/sFits on one cardFits on one cardNeeds two cardsDoesn't fit6 W$20$450–600
Tesla P10016 GB732 GB/sFits on one cardFits on one cardNeeds two cardsDoesn't fit28 W$79$180–260
RTX 4060 Ti 16GB16 GB288 GB/sFits on one cardFits on one cardNeeds two cardsDoesn't fit7 W$36$650–750
Tesla T416 GB320 GB/sFits on one cardFits on one cardNeeds two cardsDoesn't fit10 W$25$700–1,000
RTX A400016 GB448 GB/sFits on one cardFits on one cardNeeds two cardsDoesn't fit10 W$36$850–1,100
Tesla P4024 GB346 GB/sFits on one cardFits on one cardFits on one cardNeeds two cards10 W$54$350–450
RX 7900 XTX24 GB960 GB/sFits on one cardFits on one cardFits on one cardNeeds two cards15 W$77$1,120–1,310
RTX 309024 GB936 GB/sFits on one cardFits on one cardFits on one cardNeeds two cards25 W$91$1,500–1,700
RTX 409024 GB1008 GB/sFits on one cardFits on one cardFits on one cardNeeds two cards20 W$99$3,250–3,550
MI50 32GB32 GB1024 GB/sFits on one cardFits on one cardFits on one cardNeeds two cards20 W$76$630–900

Full circle: the model fits on one card at 4-bit. Half: it needs two. Power a year is at $0.18/kWh, on all day and mostly idle. Guide prices are eBay US asking prices checked 15 November 2026.

What to buy

For a first card, the RTX 3060 12GB. It holds a 14B model, it runs CUDA, so everything supports it, and it idles at about 13 W. Make sure the listing says 12GB. There's an 8 GB version with a narrower bus and it's a worse card for this.

If you need 24 GB and don't have $1,500, the Tesla P40 is still the cheap way in, at $350 to $450. Know what you're getting. It's passively cooled and needs a fan strapped to it outside a server. It takes an 8-pin CPU lead, not a PCIe one. And it's Pascal: Nvidia's 580 drivers are its last and CUDA 13 has already dropped it. llama.cpp still runs it well. Every update it misses makes it a bit more work to keep going.

The RTX 3090 is what most people end up with. It has 24 GB of fast memory and it runs CUDA. In late 2026, used ones are asking more than the $1,499 the card launched at. The RX 7900 XTX has the same 24 GB and nearly the same bandwidth for a few hundred less, and llama.cpp and Ollama are fine on ROCm or Vulkan now. If you don't need CUDA for anything else, it's the better buy.

The MI50 32GB was the bargain of 2025. Asking prices have more than doubled since the summer and ROCm has dropped it. It's also passive like the P40. Not at $700.

Skip the 3080 for this. 10 GB is too small for 14B with any context, and it pulls 320 W.

Idle power is the bill you forget

A box that runs Ollama sits idle most of the day. A 3090 idles around 25 W, which is about $40 a year at 18 cents before you've asked it anything. Two of them in an old dual-Xeon workstation idling at 90 W is a $200-a-year habit. If the machine only needs to be on when you're using it, turn it off. If it has to be on all the time, the idle figure in the table matters more than the load figure.

Tick Local AI in the planner to price a card with a host machine, or tick "The card goes in a PC I have" to rank cards on their own.

Guide prices checked 15 November 2026. Watts are typical readings at the wall; yours will vary.