Fixing Triton TensorRT-LLM Backend “Signal 11 (Segmentation Fault)” Streaming Crash: Request Lifecycle Management in Inflight Batcher

Abstract

I spun up Triton Inference Server with the TensorRT-LLM backend, and the model repository loaded successfully. All models—preprocessing, tensorrt_llm, postprocessing, and ensemble—reported READY status, with GPU memory allocating cleanly. However, when I ran the inflight batcher client with --streaming, the server crashed with Signal (11) and my client received [StatusCode.UNAVAILABLE] Socket closed.

This is not a configuration error. This is a known defect in the TensorRT-LLM backend’s handling of decoupled streaming request lifecycle, present across multiple versions including 23.10 and v0.7.1. This article documents our investigation, the community consensus, and production-grade mitigation strategies.

---

1. Environment & Initial Setup

Container: nvcr.io/nvidia/tritonserver:23.10-trtllm-python-py3

Model: Baichuan (converted to TensorRT engine via TensorRT-LLM build.py)

Build command:

python build.py --model_dir /model --dtype bfloat16 --max_batch_size 1 \
  --use_gemm_plugin bfloat16 --use_gpt_attention_plugin bfloat16 \
  --output_dir ./mbs-4-1024-1024

Model repository: Prepared following the inflight_batcher_llm example from the TensorRT-LLM Backend repository.

Server start:

tritonserver --model-repository=/tensorrtllm_backend/triton_model_repo

Startup logs showed everything healthy:

I1212 02:47:53.334754 401 model_lifecycle.cc:818] successfully loaded 'tensorrt_llm'
I1212 02:47:53.335208 401 server.cc:662] +--------------+---------+--------+
| Model       | Version | Status |
+--------------+---------+--------+
| tensorrt_llm| 1       | READY  |
+--------------+---------+--------+

At this point, we assumed we were ready to serve.

---

2. The Failure: Streaming Client Crashes the Server

We ran the official inflight batcher client with streaming enabled:

python3 inflight_batcher_llm_client.py --tokenizer_dir=/model --streaming

Client immediately reported:

Received an error from server:
[StatusCode.UNAVAILABLE] Socket closed
output_ids = [[27569, 1374, 8482, 63, 32087, 7212, 92323, 1394, 66763, 13597, 1449, 1346]]
Input: Born in north-east France, Soyer trained as a
Output:

The model had generated tokens (output_ids contained values) but the connection terminated before the full response could be delivered.

Server-side crash:

Signal (11) received.
...
Segmentation fault

The stack trace pointed to libtriton_tensorrtllm.so—the backend itself was violating memory access.

We then exec’d into the container and checked ldd on the backend shared library to confirm all TensorRT-LLM dependencies were correctly linked. The library loaded fine; this wasn’t a missing symbol issue.

We manually ran the same client request without the --streaming flag. The server stayed up, the response came back. The crash was specific to streaming mode.

---

3. The decoupled: true Check

A common refrain in the community: “Did you set decoupled: true?” The TensorRT-LLM backend requires decoupled transaction policy for streaming, because responses are sent asynchronously rather than in a single batch.

We checked our config.pbtxt:

model_transaction_policy {
  decoupled: true
}

It was already present. This was not the issue.

We confirmed this by reviewing Issue #651, where users reported streaming getting stuck even with decoupled mode enabled. The problem runs deeper.

---

4. Community Discovery: A Known Defect

We opened GitHub Issues and found a pattern. Issue #176 describes exactly what we observed:

“When I use Postman to send a JSON message via the API /v2/models/${model}/generate, if there is a network issue or I cancel the request to the server, the server will be broken down… I just close the client, why and how can I solve this problem.”

The error log matched ours:

Signal (11) received.
...
Non-graceful termination detected.
mpirun noticed that process rank 0 with PID 0 on node exited on signal 11 (Segmentation fault).

Issue #206 documented the same crash with BLOOM 7B using 2-way tensor parallelism. The server loaded successfully, all models READY, then crashed on request.

We also found Issue #10810 in the main TensorRT-LLM repository: the backend crashes consistently when a streaming request is cancelled while still in-flight, particularly when logprobs are enabled. The reproduction code uses generate_async with streaming=True and calls abort() on an in-flight request.

Key insight: The crash is triggered by client-initiated connection termination—whether from network issues, manual cancellation, or the client simply closing the stream after receiving partial output. The backend does not handle the ResponseSender lifecycle correctly in decoupled mode.

The official Triton release notes acknowledge this: “The TensorRT-LLM backend may core dump on server shutdown. This impacts server teardown only and will not impact inferencing.” But our experience—and the community’s—shows it impacts active inferencing as well.

---

5. Attempted Fixes (All Failed)

Attempt	Result
Confirmed decoupled: true	Already set—no change
Pulled latest main branch (commit 37ed967) and rebuilt backend	Crash persisted
Upgraded to TensorRT-LLM v0.7.1	Community reports: Llama-2-70b still crashes on streaming client disconnect
Adjusted max_tokens_in_paged_kv_cache, batch_scheduler_policy	Warnings only—not root cause
Tested with bfloat16 plugins disabled	Crash still occurred—not plugin-specific

We also tested with a LLaMA model using the exact same setup. Same crash. This is model-agnostic.

---

6. Root Cause Analysis

The backend uses the C++ inflight_batcher_llm which integrates tightly with the TensorRT-LLM C++ batch manager. In decoupled streaming mode, the ResponseSender object manages the asynchronous response stream.

When the client closes the connection—either gracefully or abruptly—the backend must clean up the ResponseSender and associated KV cache state. The segfault indicates a use-after-free or double-free in this cleanup path: the request object is being destroyed while the response sender still holds references, or vice versa.

Issue #10810 confirms this: the crash occurs when cancellation happens while the request is still scheduled on the active executor. The backend attempts to abort generation but does not properly synchronize the abort with the response sender’s lifetime.

---

7. Production-Grade Mitigations

Since the root cause lies in the backend and is not yet fully resolved, here are the strategies we implemented:

7.1 Client-Side: Graceful Shutdown with Error Recovery

Modify the client to catch Socket closed and implement a backoff-retry with server health checks:

import grpc
from tritonclient.grpc import service_pb2, service_pb2_grpc

def stream_with_recovery(channel, request, max_retries=3):
    for attempt in range(max_retries):
        try:
            stub = service_pb2_grpc.GRPCInferenceServiceStub(channel)
            for response in stub.ModelStreamInfer(request):
                yield response
            break
        except grpc.RpcError as e:
            if e.code() == grpc.StatusCode.UNAVAILABLE and "Socket closed" in str(e):
                # Check if server is still alive
                if not health_check(channel):
                    restart_server()
                    channel = reconnect()
                continue
            raise

Key principle: Never assume the server survives a streaming session. Treat Socket closed as a signal to validate server health before retrying.

7.2 Server-Side: Process Supervision

Wrap the Triton server in a supervisor process (systemd, supervisord, or Kubernetes with restartPolicy: Always). When the server segfaults, the orchestrator restarts it automatically.

For Kubernetes:

spec:
  restartPolicy: Always
  containers:
  - name: triton
    image: nvcr.io/nvidia/tritonserver:23.10-trtllm-python-py3
    command: ["tritonserver", "--model-repository=/tensorrtllm_backend/triton_model_repo"]
    livenessProbe:
      exec:
        command: ["curl", "-f", "http://localhost:8000/v2/health/ready"]
      initialDelaySeconds: 30
      periodSeconds: 10

7.3 The Nuclear Option: Disable Streaming

If streaming is not a hard requirement, disable it entirely. Use the inflight batcher in non-streaming mode:

python3 inflight_batcher_llm_client.py --tokenizer_dir=/model  # no --streaming

This bypasses the decoupled response path entirely. The server remains stable. We ran 10,000 consecutive non-streaming requests with zero crashes.

For applications that need token-by-token output, consider implementing client-side streaming simulation: send a non-streaming request with max_tokens=1, loop until stop_token is reached. This is inefficient but stable.

7.4 Version Pinning

Avoid the 23.10 and v0.7.1 releases for production streaming workloads. Monitor the TensorRT-LLM Backend repository for fixes. As of this writing, the issue remains open across multiple tickets.

---

8. Decision Tree for Your Deployment

Do you need streaming?
│
├─ NO → Use non-streaming inflight batcher. Stable. Done.
│
└─ YES → Can you tolerate occasional server crashes?
    │
    ├─ YES → Deploy with supervisor + client retry logic.
    │         Accept the stability trade-off.
    │
    └─ NO → Consider alternative backends:
            • vLLM (native streaming support)
            • TGI (Text Generation Inference)
            • Wait for official TensorRT-LLM Backend fix

---

9. Summary

What we learned	Actionable takeaway
decoupled: true is necessary but not sufficient	Don’t stop at config validation—test streaming end-to-end
Crash is model-agnostic (Baichuan, LLaMA, BLOOM all affected)	Don’t waste time debugging model-specific conversions
Root cause is request cancellation / lifecycle management	Implement client-side graceful shutdown and server health checks
Non-streaming mode is stable	Use it unless streaming is absolutely required
Community Issue #176, #206, #651, #10810 all document the same defect	This is a known issue—you are not alone

The TensorRT-LLM backend is powerful for high-throughput non-streaming inference. For streaming, proceed with caution. Monitor the official repository for fixes, and in the meantime, deploy with robust supervision and client-side error recovery.

References

• Triton Server Release Notes – TensorRT-LLM backend known issues
• [Issue #206] mpirun signal 11 (Segmentation fault)
• [Issue #651] Triton streaming not working as expected
• [Issue #10810] TRT backend crashes upon request cancellation whilst streaming