Local inference on hardware people actually own
The questionHow much use can you get from a model small enough to sit on your own premises?
What is running
- · Hugin and Munin, two public endpoints running on a Mac mini M4 in Norway, free and without sign-in.
- · A retrieval layer around the model that does more for answer quality than the parameter count does: topic routing with local embeddings, and examples picked from what users actually gave a thumbs up.
What is still open
A 4-bit quantised model at four billion parameters loses to the cloud models on most things. The open question is where it is nonetheless good enough, and how much of the gap can be closed with retrieval and context rather than larger weights.