Jan Sverre Bauge

I Tested Ternary Bonsai 27B Locally on an RTX 4090

My Ternary Bonsai 27B test on an RTX 4090 reached 83-85 tokens/s with 7.8 GB extra VRAM, solid JSON, document work and tool calling - plus clear limitations.

Ternary Bonsai 27B running locally on an RTX 4090
Ternary Bonsai 27B compressed 27 billion parameters into a roughly 7.2 GB local model.

Ternary Bonsai 27B squeezes 27 billion parameters into a model file of about 7.2 GB. On my RTX 4090, it delivered 83-85 tokens per second while retaining enough practical ability to do far more than ordinary chat.

The model and runtime used about 7.7-7.8 GB of additional VRAM above the machine’s idle level. More importantly, it produced valid JSON, extracted documented facts with verifiable text evidence from a long document, called tools, read and wrote files, created short-video scripts and used an external image service through the host machine.

This is no miracle model, though. English was clearly better than Norwegian, the OCR test was only partly successful, and DSpark brought no speed gain on my setup. What matters is how much real work survived the brutal compression – and how useful such a small 27B model can actually be when it runs locally.

What is Ternary Bonsai 27B?

Ternary Bonsai 27B is a compressed version of a Qwen3.6-27B-based model whose weights mainly have three possible values: -1, 0 or +1. Prism ML lists a distributed size of about 7.2 GB, compared with roughly 54 GB for the FP16 version. It is released under the Apache 2.0 license.

Put simply, model weights are all the small numbers the model learned during training. Conventional models store them using many more possible values. Ternary restricts the language component to three: negative, zero or positive. That dramatically reduces the storage requirement, but it also needs a format and compute kernels that understand how those three values are packed.

That last point is crucial. The file may be called GGUF and Q2_0, but this is not the same as an arbitrary conventional 2-bit quantization that you can feed into any GGUF-compatible application. The recommended file, Ternary-Bonsai-27B-Q2_0.gguf, uses groups of 128 weights and Prism’s own low-bit kernels. The model should not be confused with the smaller 1-bit variant or the separate DSpark files.

Prism claims that the ternary version retains about 95 percent of the FP16 result across its test suite. That is interesting, but it was not my yardstick here. I wanted to find out whether the model could do work I actually need on my own machine, without letting a benchmark table write the conclusion in advance.

How I ran Ternary Bonsai 27B on Windows

BF16 would not load through Ollama in my setup, so I moved on to Prism ML’s own llama.cpp fork with CUDA 12.4. The native build worked. The relevant Prism release provides prebuilt Windows files, while the Bonsai demo documents the Windows setup, server and model selection.

There is another small trap here. The release link using the filename llama-prism-b9591-62061f9-bin-win-cuda-12.4-x64.zip returned a 404 during my test. The actual binary package was named llama-prism-b1-62061f9-bin-win-cuda-12.4-x64.zip. I also needed the CUDA package cudart-llama-bin-win-cuda-12.4-x64.zip, which includes cudart64_12.dll, cublas64_12.dll and cublasLt64_12.dll.

Here is the short PowerShell route using a neutral local directory. Download the two ZIP files from the release page, put them in C:\AI\bonsai\downloads, and fetch the model file from Hugging Face:

$Root = "C:\AI\bonsai"
New-Item -ItemType Directory -Force "$Root\bin", "$Root\models" | Out-Null
Expand-Archive "$Root\downloads\llama-prism-b1-62061f9-bin-win-cuda-12.4-x64.zip" -DestinationPath "$Root\bin"
Expand-Archive "$Root\downloads\cudart-llama-bin-win-cuda-12.4-x64.zip" -DestinationPath "$Root\bin"
hf download prism-ml/Ternary-Bonsai-27B-gguf `
  Ternary-Bonsai-27B-Q2_0.gguf `
  --local-dir "$Root\models"

Make sure llama-server.exe and the CUDA DLL files end up in the same executable directory. The exact subdirectory structure may depend on how the ZIP files are extracted. You can then start the server locally like this:

& "C:\AI\bonsai\bin\llama-server.exe" `
  -m "C:\AI\bonsai\models\Ternary-Bonsai-27B-Q2_0.gguf" `
  -ngl 99 -c 8192 --host 127.0.0.1 --port 8080

The server exposes an OpenAI-compatible endpoint at http://127.0.0.1:8080/v1. Two CLI details will also save you some time. This build did not accept llama-cli -i; conversation mode had to be started with -cnv. For a single task, llama-completion was the better choice. llama-server -sys was not supported either, so I had to send the system message in the API call.

At the time of my test, Prism recommended a temperature of 0.7, top-p of 0.95 and top-k of 20. The model can run with reasoning enabled or disabled, and it supports a separate reasoning budget. I disabled reasoning for the work tests to get cleaner, more predictable output. Visible thinking can help with debugging in chat, but it quickly turns into a lot of English meta commentary before the actual answer.

How fast was it, and how much VRAM did it use?

On the RTX 4090, I usually measured 83-85 tokens per second during generation. llama-bench produced about 84.7 tokens per second on tg128, while the first short completion test ran at about 68 tokens per second after roughly four seconds of loading. That is responsive enough for the model to feel fast in ordinary use.

The machine used about 1,720 MiB of VRAM at idle. With the server and model loaded, usage rose to around 9,470 MiB. The difference is approximately 7.7-7.8 GB for the model, KV cache and runtime with an 8,192-token context. VRAM usage remained broadly flat at about 9.5 GB while the server was running.

That leaves a great deal of headroom on a card with 24 GB of VRAM. It does not mean everyone should expect the same speed: my result applies to Windows, this Prism build, an RTX 4090, full GPU offload and the specific Q2_0 file. Raw token speed should never be read without the hardware and setup beside it. But the size is not merely a number on disk. The model was genuinely small enough to share the machine with other applications without forcing me to clear everything else out of GPU memory.

This is a different category from conventional 27B models at four bits or more. The specialized format requires the correct runtime, but the payoff is that a 27B model can sit at roughly 9.5 GB of total VRAM usage on the machine and still generate text very responsively.

English was good, but Norwegian was not ready to publish

My regular chat test starts in English with a request for help on an article about the history of AI from 1970 to 1990. Bonsai understood the period, proposed a tidy structure, identified relevant themes and suggested concrete next steps. The answer was friendly and practical, with generation running at around 83 tokens per second.

In the next round, I provided context about a Norwegian technology site, my interest in AI since 1996 and why I use the term AI. The model understood that it should switch languages. The execution was the problem: it mixed Norwegian and English, introduced Swedish-looking forms and produced words that do not belong anywhere. The attempt was pleasant enough, but the text was nowhere near publishable without heavy editing.

That is an important limitation for Norwegian users. The model can understand Norwegian context and answer in Norwegian, but my test does not support recommending it as a finished Norwegian writer. I would use English as the working language and treat Norwegian output as raw material. I did not systematically test longer and more difficult Norwegian tasks.

JSON and document work were the clear strengths

The first structured task was to create an article package as strict JSON. The result parsed directly, contained every required key, and included six outline items and five claims. The task took about 17.7 seconds and produced 1,455 completion tokens. This is exactly the kind of boring, useful work a local model needs to master if it is going to be more than a chat demo.

I then gave the model the complete text of a technical article about BitNet b1.58. With 8,368 prompt tokens and 2,687 output tokens, the extraction took about 37 seconds. The model returned 12 facts, 10 talking points, six caveats and eight glossary items as valid JSON. I had to increase the context from 8,192 to 16,384 because the document was otherwise too long.

The most convincing test required every fact to include documentary text evidence and a location. The result contained 14 facts, and all 14 quoted fragments could be found in the source document. A simpler variant achieved 13 out of 13 matches. That does not prove the model never hallucinates, but this particular test demonstrated solid document grounding when the task and schema were tightly specified.

This is where Bonsai became more interesting than it was in ordinary chat. It processed a long document quickly, followed a structure and produced a machine-readable result that I could verify automatically. That has real value in local workflows for document extraction, sorting and preparing material.

Tool calling worked – but the host did the work

Through the server’s OpenAI-compatible API, I first gave the model access to a simple get_time tool. It returned the correct tool call with an empty argument object, the host fetched the local time, and the model used the result in the second round. The same pattern worked with read_file and write_file.

In one file test, the model asked the host to read a document and write ten points to a JSON file. The complete chain took about 15.6 seconds and ended with valid output. In another task, it read the research notes and wrote five English scripts for short videos in about 14.9 seconds. The scripts were slightly shorter than the model itself claimed, but the result was still very good.

I also tested image generation by exposing a generate_image tool that called Kie z-image on the host machine. Bonsai read the research, created a tool call for a 16:9 image and received the file result after about 52.5 seconds. The model did not generate the image itself. It selected and called an external tool, while the host handled the service and API key.

Article image generated after Ternary Bonsai 27B read the test research
Ternary Bonsai 27B requested this 16:9 image through an external generate_image tool on the host machine.

That distinction matters for both practical and security reasons. Bonsai does not have built-in internet access, file access or image generation. The application around the model decides which tools it may request. The API key remained in the user’s session on the host machine and was never placed in the model prompt.

Vision understood the scene, but OCR was only partly reliable

Vision required a separate multimodal projector and llama-mtmd-cli. Prism publishes projector files of roughly 0.6-0.9 GB in addition to the language model. Using the Q8_0 projector and at least 1,024 image tokens, the simple tests took about five to seven seconds.

On a simple picture of an apple, the model identified the subject correctly. It is a modest test, but it confirms that the vision chain was connected correctly and that the model was not merely answering from the filename or prompt.

Apple image that Ternary Bonsai 27B identified in the simple vision test
I used the apple image to verify that the vision chain and separate multimodal projector were working.

The demanding test used test.png, a poster featuring a man in a green hoodie, a laptop, a brain visualization, holographic elements and a great deal of Norwegian text. The model described the scene well and captured the large headline approximately. It also identified the Claude and J-space Research logos.

Norwegian AI poster used to test OCR and image understanding with Ternary Bonsai 27B
I used this Norwegian AI poster to test scene understanding, logo recognition and exact text reading.

As an OCR tool, it was much less reliable. It interpreted “AI” as “Al” and misread Norwegian words. My verdict is therefore that the model can understand what a poster is about, but it should not be trusted as the authority for exact transcription. It handled the scene and broad context; quotations and small text still need manual checking.

DSpark worked, but made the model much slower for me

DSpark is a separate draft model that proposes several tokens before the main model checks them. The idea is to reach the same result faster because the main model can approve several suggestions in one pass. Prism provides a Q4_1 drafter of about 2 GB and reports speed gains in its own CUDA measurements.

The technical mechanism worked in my test. The API run showed 216 proposed and 144 accepted tokens, or about 67 percent. In llama-speculative-simple, the acceptance rate was around 82 percent. Speed was the problem: the ordinary API run reached about 84 tokens per second, while the API with DSpark fell to roughly 5.5. The simpler speculative client managed around 20.

This may depend on the build, workload and settings. Prism’s measured gains on other setups are not necessarily wrong, but they did not appear on my RTX 4090 in this test. DSpark also had to run with one parallel stream, a larger context and a new prefill. I would therefore treat the feature as experimental until a specific workload demonstrates an actual benefit.

Is Ternary Bonsai 27B worth the effort?

Yes, if the goal is to fit as much practical model capacity as possible into a small memory footprint. About 7.2 GB on disk, roughly 7.8 GB of additional VRAM and 83-85 tokens per second made for an excellent combination on my machine. JSON, document grounding, file work and tool calling made the model relevant to real workflows.

No, if what you primarily want is a good Norwegian chat model. Norwegian was the clearest weakness. Vision worked, but the OCR result needed checking, and DSpark caused a severe slowdown instead of an acceleration on my setup.

For me, this was a successful test. Not because the model beat a particular benchmark, but because it processed documents, wrote verifiable JSON, used tools, created video scripts and had a host generate an article image. Ternary Bonsai 27B shows that a 27B model of around 7 GB can be far more than a technical curiosity.

Frequently asked questions

What could Ternary Bonsai 27B do in practice?

It produced valid JSON, extracted documented facts from long texts, called tools, read and wrote files, and created short-video scripts. Through tools on the host machine, it could also request image generation from an external service.

How much VRAM does Ternary Bonsai 27B need?

With an 8,192-token context, the model and runtime used about 7.7-7.8 GB of additional VRAM in my test. Total machine usage was around 9.5 GB while the server was loaded. Requirements will vary with context, backend and other applications.

Is Ternary Bonsai 27B good at Norwegian?

Not in my chat test. The model understood Norwegian context, but mixed languages and produced several strange words. English was clearly better. I would treat its Norwegian output as raw material that needs thorough editing.

Can Ternary Bonsai 27B read and generate images?

It can interpret images using a separate vision projector. New images in my test came from a tool call to an external service on the host machine, not from Bonsai alone. The API key remained outside the model prompt.

Leave a Reply

Your email address will not be published. Required fields are marked *