All tools / NVIDIA Nemotron 3.5 Bolt E4B

Tool profile

NVIDIA Nemotron 3.5 Bolt E4B: Local GGUF Model and Memory Checks

NVIDIA Nemotron 3.5 Bolt E4B is a text-in, text-out model with a Q4_K_M GGUF release for local use. The 4GB repository name is not a guarantee that the whole application fits in 4GB of RAM: budget for the runtime, context cache and other processes too.

Official access and docs

Checked October 25, 2026 · Official-source review; no hands-on benchmark

What is actually downloadable

The NVIDIA model card lists 8B total parameters, 4B effective, Q4_K_M precision and text input/output. The repository contains Nemotron-3.5-Bolt-E4B-Q4_K_M.gguf. NVIDIA lists OpenMDW-1.1 terms; check that license for your intended use.

This is distinct from the BF16 version and the optional DFlash drafter. A voice-assistant use case still needs separate speech input/output components; the text model alone does not record or speak audio.

Use the pinned recipe as the compatibility reference

The card's quick start uses ghcr.io/ggml-org/llama.cpp:server-cuda13-b11459, the named GGUF file and --jinja. Its recipe gives separate commands with and without the drafter. Start without the drafter to reduce the number of moving parts.

The summary still says the deployment recipe is pending even though commands are present. Treat the published commands as a starting point to validate, not a guarantee for every existing llama.cpp build or GPU. We have not downloaded the weights or run inference.

Check memory before maximizing context

NVIDIA lists input and output lengths up to 64K tokens in the model summary. That is a model capability description, not a measured memory budget for your machine or a promise of 128K combined context.

  1. Record your runtime version, quantization and available system/GPU memory.
  2. Load the base model with a short context and a small output limit first.
  3. Confirm the response is readable and the intended template is active.
  4. Increase context gradually while watching peak memory and response time.
  5. Add the separate drafter only after the base path works, then measure whether it helps your workload.

If loading fails, check architecture support, file completeness and the runtime log before assuming the model is broken. If memory runs out, reduce the context or revise offload settings for the supported runtime.

A useful first local evaluation

Use a few prompts with known answers and a small tool schema you control. Compare outputs with thinking enabled and disabled, checking the model card's template controls. Keep the same prompt, context, hardware and output limit when comparing latency.

Download through the official file list. The weights are available without an AITF fee; compute, storage, electricity and any commercial hosting are separate costs.