NVIDIA/TensorRT-LLM v1.4.0rc1
Disaggregated serving with the default Python KV cache transceiver may fail generation requests with a KV cache transfer error after streaming requests are cancelled (observed on A100).
ProofCode ↗
Key points
- The OpenAI JSON-schema serving example (examples/serve/openaicompletionclientjsonschema.py) may return an unbounded response with Qwen3 models.
- Add Qwen3.8-Flash-Next NVFP4 as an experimental Modeling V2 target #20143
- Default to KV cache manager V2 and the Python KV cache transceiver (BREAKING) #17495
- Integrate the Mooncake connector with KV cache manager V2 #19171
Sources (1)
- [1]NVIDIA/TensorRT-LLM v1.4.0rc1GitHub: NVIDIA/TensorRT-LLM · Oct 22, 01:38 AM
- Disaggregated serving with the default Python KV cache transceiver may fail generation requests with a KV cache transfer error after streaming requests are cancelled (observed on A100).
- The OpenAI JSON-schema serving example (examples/serve/openai_completion_client_json_schema.py) may return an unbounded response with Qwen3 models.
Extractive summary: sentences quoted from the sources.
Before this
- Oct 21, 2026Recursive shows first result of its AI-driven research process with Flint
- Oct 20, 2026Voxtral 3
- Oct 20, 2026[AINews] DeepSeek DeepInfer: Open Source SOTA Inference running 155T toks/day/cluster making $22m/day at ~80% margin for a 81 day payback
- Oct 20, 2026Do Diffusion LLMs Scale? Testing Mercury 3 + Decide
- Oct 20, 2026MPS in PyTorch 2.15
- Oct 19, 2026Amazon SageMaker JumpStart: 2026 year-to-date in review
