My Journey to Local LLM

My first real exposure to LLMs was through work — I was chosen for the GitHub Copilot pilot group. Around the same time, my existing JetBrains All Products Pack subscription happened to include AI usage that basically felt unlimited. At that point, LLMs weren't really doing tool calling — it was just a "smart" chat bot. I found the limitations on both sides quickly: only useful in very specific, well-documented scenarios with limited context windows. In other words, only useful for very small parts of code at a time.

By Fall 2025, we were exposed to LLM agents in GitHub Copilot for application modernization. My employer wouldn't whitelist anything past GPT-4.1/4o, so basically useless. My JetBrains AI allowance vanished quickly as the company switched to token-based billing. February 2026 changed things. We finally got access to Claude Sonnet 4.5 at work, and I subscribed to a personal GitHub Copilot license. I also took a one-month stab at Claude Code's $20 plan — burn through that, then fall back to Copilot prompts. In an agentic loop, that meant a solid amount of usage for $40/month. I was willing to do that for a few months at the time.

There was a lot of advice being thrown around, and a lot of it was wrong. Too many influencers aren't engineers and haven't deep-dived into the technology. The result: noise. Lots of it. Few people actually know what they're doing with LLM agents, or even know what to do with them. Even Claude Code — the gold standard at the time — was a buggy mess. It would lose context, hallucinate file paths, and occasionally delete half your working directory because it misread its own output. I have the git log to prove it.

LLM coding agents are impressive. They have a lack of vision and knowledge that they can make up for by basically running spikes and verification tasks in seconds, letting them gain some idea of vision back. Sometimes they get stubborn, and you have to help them. A good engineer with a good BS filter can basically ignore the LLM's BS, get to the planning, execution, and verification, and really extend their capabilities. Without the BS filter, you take what the AI says literally. Without being a good engineer, it's hard to know when the LLM is making bad choices. It's much like a calculator in math. If you fully understand a concept, a calculator accelerates you. If you don't understand the concept, you may finish with a calculator, but you'll still walk out confused and not exactly confident.

A single prompt could do a whole sprint's worth of work. Granted, it took some of the fun away. Many developers began lamenting that LLMs were taking their favorite part of the job away — the coding. Most engineers began as program hackers, writing small programs that did small things, and feeling the joy of making something work. Then we get paid to do it, and we start pairing that with customers and their hard business rules, paperwork, oversight, real financial risks, career growth concerns, etc. Most developers dislike the work part of the job, but love the programming part. I'm no different — I don't enjoy the routine paperwork, the scrum ceremonies (even though I'm a scrum master), the very difficult and conflicting business rules, etc. But I wasn't going to let LLMs ruin what fun I do have with my career. Interacting with LLMs themselves isn't too different than writing a program: you have a goal, and you're writing a form of code to get that goal done. It's just that the "compiler" — in a manner of speaking — is non-deterministic but also very powerful.

So I found a new game. I got into tweaking agent instructions to see how far a single prompt could go without losing quality. Not "prompt engineering" — more like being as lazy as possible. My goal has been to let me be as vague and short as possible in a conversation, and let the LLM figure out what I want based on the core instructions I've built up. I built an agentic workflow that I brought into work and showed to my whole team — with relatively good success.

But then Claude Code's usage limits shrank, and GitHub Copilot started rate limiting. I cancelled Claude Code, bumped GitHub Copilot to Pro+ ($40/month). That removed rate limiting for about a month, then Microsoft announced they were dropping the prompt billing strategy entirely and would charge API rates for token usage. A single prompt could do half a day's work, burning thousands of tokens. With the new pricing, my personal GitHub Copilot bill was going to be $300+/month if I continued. Work had the same problem, but they didn't seem to care enough to do anything about it. Here's the thing: cloud AI has the same benefits and problems as cloud hosting. Someone else manages the infrastructure. You pay for what you use. It just works — until it doesn't. Until they change pricing. Until they limit your usage. Until they decide you don't have full rights to what you're doing with it. Until the guardrails (Mythos, Fable) tell you what you can and can't ask. You trade convenience for control. You take on the operational burden. You make it stable yourself.

As an engineer, that trade-off is familiar. Back at Smart Furniture, I was the main developer and server admin — effectively the webmaster for a multi-million dollar e-commerce business. (Granted, they went bankrupt after I left, but the business they were in was hard.) Self-hosting was the norm. Most businesses have "self-hosted" their technology for decades, even if it wasn't on-site — they still managed their DNS, their router, their web server, their database. I brought that into my personal life too. Cloud AI is good enough for most people, and I get why. But I've always disliked the lack of ownership, accessibility, and freedom that apps and cloud services impose — especially on the "smart" side of things. I occasionally use apps (probably YouTube the most), but everything-is-an-app, everything-is-cloud, everything-is-someone-else's-problem is a model I've tried to opt out of. Cloud AI fits right next to social media as a consumption product. It just doesn't pair well with me.

Beyond convenience, there's a risk question on both sides. Just because a model is downloadable and free doesn't mean it won't go stale - newer versions may never get published. That's a risk with self-hosting. But I do swear there are days when the AI from GitHub Copilot at work is worse than the one from yesterday, even with the same name supposedly powering it. So stability and quality are not guaranteed on either side — you just get to control which risks you take.

Someone might say "but electricity", but I live under the TVA area, so rates are pretty low, so power usage wasn't a material cost to truly factor in unless I bought a serious card and it used a serious amount of power constantly. I didn't actually track it carefully at first, but even being generous with the estimate, I wasn't concerned about that.

When I signed up for GitHub Copilot at home, I did some basic math. At 300 prompts/month for $20/month, it'd take ~6 years of subscriptions to zero-out a $1500 graphics card - and that's without counting electricity. Even on the $40/month sub, running multiple agents at a time, it'd still take 3 years. NVIDIA had nothing new at ~$1500 that was worth getting for running an LLM agent. The only game in town seemed to be the RTX 5090 at ~$4000 - or 8+ years vs the subscription. And I'd probably need a bigger case and a better PSU. My decision at the time was it wasn't worth it. In 3-8 years, the hardware and the AI capabilities would be completely different. Sinking $4000 into a graphics card for "hobby" stuff would've been a horrible financial decision, and a selfish one at that.

I should denote why I spent so much time weighing this decision: As a father to three young kids, with a stay-at-home mom, I'm the primary breadwinner, and my attitude is that everything I make monetarily wise belongs to my family first. I owe my wife and kids a roof, utilities, schooling, food, etc, but also I owe them memorable Christmas's, birthday's, etc. Yes, we budget, try to be frugal, but the largest ticket financial item for our family is housing. Our housing is too much of my paycheck, and there's no route to return to pre-COVID housing prices, so I weight every dollar I spend so I know we can buy food and pay rent. Luckily, we've been working on an exit plan from renting, but every dollar I spend on an AI-hobby is a dollar I risk not being able to pay off a credit card, and I will be beyond ashamed if I have to do that - especially for something that isn't producing my family money. I'm salaried, and my at-home AI will not help my salary go up.

Rough breakdown:

At that point, local didn't make sense. But I was also looking at cards, just in case. Rough GPU options I was seeing:

The 12-16GB cards were obviously too small - 24GB was the minimum that made sense, especially since a quick google search confirms: Qwen3.6-35B-A3B uses ~22GB of VRAM. Last I checked, 22GB was less than 24GB, and that's all this card would be doing. Beyond that, $4,000 for the 5090 wasn't a call I could justify. Based on what I say, the 3090 made the most sense, even used. If I bought two, and I could somehow split processing between the two, I then would have 48GB for $3000 vs the 5090's 32GB for $4000. But I wasn't willing to drop $1500 on a maybe. I compared stores and found the best return option to be via Amazon's refurbished price. Granted, the free shipping and 90-day return policy came with an upfront fee, but I'd get to avoid the sunk cost fallacy — if it didn't work out, I could walk away.

Then the pricing shift hit. It'd now take only 3-4 months of API pricing through GitHub Copilot to pay off a graphics card, and the other subscriptions had too many rate limits going on. The numbers flipped in about two weeks' time - GitHub Copilot was going away as a viable personal tool, OpenAI and Anthropic made it clear you don't have full rights to their AI, and Qwen3.6 27B/35B-MOE had released, bringing good performance to consumer hardware. I didn't know about the R9700 yet. At that point, the "conventional wisdom" on the internet pointed to the RTX 3090 as the card to get. So that's what I ordered - same day, basically. The 9700, the ROCm improvements, the $33/GB math - all of that came later, after I hit the 3090's limitations and started looking harder.

I want to own my models and my data — no one revoking my access because I asked the wrong question or violated a policy I didn't know about. I want to run whatever model I want, at whatever quant I want, with whatever context window I want — no whitelists, no rate limits, no "our lawyers said no." My prompts, my code, my conversations about my life — I want them to stay in my house. No telemetry, no training on my data, no third-party access. The trade-off is I have to make it stable myself. And that's a problem I'm willing to solve.

So, shortly after GHCP announced their usage-based billing changes on April 27th, I ordered the 3090. I made my decision. Part 2 covers the hardware side — the 3090, the fan noise, the VRAM wall, the R9700, Docker, Vulkan, and where things stand today. I'll link it when it's done.