Local Models 1: Everything Passed, Nothing Worked

Watching tokens stream out of a model running on your own hardware hits differently than watching them come back from an API call. The text is the same, but one is a request and the other is something you put together yourself.

I have hardware that can run a genuinely useful model, and a Claude subscription whose ceiling I rarely touch in my workflows. Running something sized to my actual needs is cheaper, but the bigger draw is what I'll have to learn to make it work: how a model behaves inside a harness, which of my workloads actually need a large context window, and how far a model can be squeezed onto a single GPU before it stops being worth using. The goal is a setup I rely on every day, so it has to hold up on real work.

The setup

• Model: Qwen 3.8-27B, Q8_0, served by llama.cpp
• Harness: pi, running in WSL with the pi-llama package
• GPU: RTX 5090
• Context: 32,768 tokens

Every serious coding harness right now lives in a terminal, and most of them would get the job done. I used opencode for a while and liked it, and I gave Hermes a try, but neither was built around the thing I care about most here: spending as few tokens as possible on anything that isn't the task. Pi is the most popular of the stripped-down harnesses. It's small by default, easy to extend, and I can shape it around how I work instead of the other way around. With a 32k window, a harness that spends less of it on its own overhead goes a long way.

The quant choice comes from the last time I ran Qwen locally. With 3.6 I had plenty of context and the raw output was good, but I could never hand it a task and trust it to finish. I also never tuned anything. This time I'm putting real effort into tuning, so I'm starting with the most capable quant that fits in VRAM and working down only if I have to. Q8 caps me at 32k tokens without spilling into system RAM. A smaller quant would buy a bigger window, but a bigger window on a model I can't delegate to doesn't solve much.

The first real task

The test: update every package in this site, a Next.js + Payload app, to the latest version without breaking anything. It's a useful stress test because it touches everything (peer ranges, breaking API changes, lint config, types), it doesn't always fail loudly, and it shows how much judgment the model brings when the prompt doesn't supply any.

Qwen worked through it and ended with a passing build. Along the way it caught a real breaking change: lottie-react v3 dropped its default export and renamed animationData to src. It found the new API by reading the package's type definitions and updated all three components:

- import Lottie from 'lottie-react'
+ import { Lottie } from 'lottie-react'

- <Lottie animationData={ScrollAnim} loop={true} className="w-10 h-10 opacity-50" />
+ <Lottie src={ScrollAnim} loop={true} className="w-10 h-10 opacity-50" />

The build passed, the types checked, and when I reviewed the diff it matched the new API exactly. I merged it into my test branch, and the site was broken.

What actually broke

I handed the broken branch to Opus in Claude Code to find the cause. The migration above was correct as far as it went, but v3 also changed its defaults. In v2, lottie-web underneath defaulted autoplay and loop to true. In v3:

// lottie-react/build/animation/useLottieAnimation.js
const { src, loop, autoplay = false, ... } = options;
const [loopValue, setLoopValue] = useState(loop ?? false);

Every animation on the site rendered its first frame and stopped. Nothing in the pipeline looked at the rendered page, so nothing noticed. The fix was one prop per component:

- <Lottie src={ScrollAnim} loop={true} className="w-10 h-10 opacity-50" />
+ <Lottie src={ScrollAnim} loop={true} autoplay className="w-10 h-10 opacity-50" />

That wasn't the only thing that passed without working:

• graphql went to ^17 while Payload declares a peer range of ^16.8.1. The install stayed quiet because .npmrc still had legacy-peer-deps=true from an old workaround, a flag that silences every peer violation. The schema still built under 17, so nothing was broken yet, but it was an unsupported combination nobody had chosen.
• ESLint 10 and TypeScript 7 both came in with the update. No published version of eslint-plugin-react, eslint-plugin-import, or eslint-plugin-jsx-a11y supports ESLint 10, and typescript-eslint throws on import under TS 7. Lint couldn't run at all.
• A peer conflict for @react-three/fiber got resolved carefully, for a package nothing in the app imports.

The ESLint detour

Before any of this surfaced, Qwen spent about twenty minutes on lint. pnpm lint failed because Next 16 removed next lint, and running ESLint directly hit a TypeError: Converting circular structure to JSON inside @eslint/eslintrc. It traced that through FlatCompat to a self-referencing plugin object in eslint-plugin-react and confirmed against the old lockfile that the issue predated the update. It was reasonable debugging on a problem that didn't need solving, and it kept running into the context ceiling:

Error: 400: request (33567 tokens) exceeds the available context size (32768 tokens)
[compaction] Compacted from 33,684 tokens

Each compaction dropped some of what it had already ruled out, so the next few turns went to rediscovering it. Long investigations and a 32k window don't mix well, and that's a tuning problem for a later post.

What this changes

"Without breaking anything" only means something if there's a way to check. This project had a build step and a lint command that couldn't run, so the model's only signal was the build, and the build passed while the site was broken. Qwen did what it could with the definition it had. Silent default changes, suppressed peer warnings, a dead lint script: a person without tests would have walked into the same gaps. The difference is that an agent won't stop to ask what you meant. It fills in whatever behavior you didn't define with a guess.

That puts two things ahead of more model tuning:

• Tests for the site. CI that runs lint and build on every pull request (today the only workflow is the deploy, so nothing checks a branch before it merges), plus a few Playwright smoke tests for the paths that matter. Playwright drives a real browser, which makes it the only layer that would have caught the frozen animations.
• A repeatable eval for the model. The same tasks with explicit pass/fail criteria, run against Q8 and later Q6, instead of judging each session by how it felt.

Both come down to the same rule: before handing work to an agent, write down what done looks like in a form the agent can check.