A local LLM on your own server: what it costs, what it needs
Two products answer customers with a language model running on a graphics card from 2016. Measured over 15 days: 12 W on average. The big card sits on the workstation, not in production. The numbers, and what they mean for a small business.
Short answer
The feature inside a product does not need an expensive card: Turboscore answers with a four-billion-parameter model running on a GeForce GTX 1070 with 8 GB, and the card drew 12 W on average over the last 15 days. The 24 GB card sits on the workstation and is used for coding agents and development; the product only uses it as a fallback.
What actually runs
The production server has a Ryzen 9 7950X3D, 61 GiB of RAM and a GTX 1070 with 8 GB. Ollama runs locally on it.
Turboscore is configured with gemma3:4b as its primary model, kept warm for 60 minutes after
the last question, and llama3.1:8b on the workstation’s card as the fallback. As I write, gemma3:4b is resident
at 3.1 GB, entirely on the GPU, with an 8192-token context. Spotscore runs its own local model through Ollama the same
way.
The model is the exception, not the rule. Most questions are read by rules, as A test that can fail describes. Since 28 September, the span the metrics store holds: 653 searches in Turboscore; the intent-analysis counter recorded 11, all resolved by rules.
What it needs
Memory on the card is what decides. The figures are from ollama list and ollama ps on the two machines:
- 4 billion parameters (
gemma3:4b): 3.3 GB on disk, 3.1 GB loaded. Fits the 8 GB card with room to spare. - 8 billion (
llama3.1:8b): 4.9 GB on disk. Fits 8 GB. - 14 billion (
qwen2.5:14b,qwen3:14b): 9.0–9.3 GB on disk, 12 GB loaded with an 8192-token context. Needs the 24 GB card; today 27 % of it sat on the CPU because the memory was shared with something else. - 26–27 billion (
gemma4:26b,gemma3:27b): 17 GB. Fills the 24 GB card, runs entirely on the GPU, around 170 tokens per second. - 32 billion: 19–20 GB on disk, and 45 % of the model lands on the CPU. Around 12 tokens per second. Not worth it.
The rest is operations: a timeout on the call (15 seconds in Spotscore’s chat), a rule-based path that answers when the model does not, and a test that can fail.
What it costs in power
Both cards report power to Prometheus every 15 seconds. The store keeps 15 days, so this is 28 September to 13 October:
| GTX 1070 (production) | RTX 4090 (workstation) | |
|---|---|---|
| Average while the machine is on | 12 W | 53 W |
| Peak | 199 W | 452 W |
| Time above 10 % load | 0.6 % | 8.8 % |
| Machine on | 100 % of the time | 70 % of the time |
| Card alone, per day | 0.3 kWh | 0.9 kWh |
Multiply the kilowatt-hours by your own electricity price. The whole machine’s draw is not measured; these are the cards alone. No purchase prices here: both are consumer cards, and the 1070 is ten years old.
What it does not solve
The workstation is off 30 % of the time. That is why it is the fallback and not the main path: a product cannot lean on a machine that is not always on. And a local model is as happy to answer wrongly as one in the cloud; the difference is that the error stays with you. The test that caught an invented brand filter in Turboscore is in the note above.
For a business here
Data does not leave the building, and there is no bill per token. What it takes is one card with enough memory for the model plus its context, Linux, and someone who reads the log. The card itself can be the one already sitting in a machine.