Streaming slows down as responses get longer

Jc-965 commented

2026-09-26 16:45:41 +00:00

Member

Every piece of text the model streams goes through a safety check for prompt leaks and secrets before it's sent on. That check re-reads the whole response so far each time, so it gets slower with every chunk, from a fraction of a millisecond at the start to about 5 ms per chunk once the answer is a few thousand characters long. It also runs in a way that holds up every other request on the same server while it works.

We want long responses to stream just as smoothly at the end as at the start, and one long response shouldn't slow anyone else down. The leak and secret protection should stay.

Every piece of text the model streams goes through a safety check for prompt leaks and secrets before it's sent on. That check re-reads the whole response so far each time, so it gets slower with every chunk, from a fraction of a millisecond at the start to about 5 ms per chunk once the answer is a few thousand characters long. It also runs in a way that holds up every other request on the same server while it works. We want long responses to stream just as smoothly at the end as at the start, and one long response shouldn't slow anyone else down. The leak and secret protection should stay.

Jc-965 referenced this issue from ScottyLabs/cmugpt-surface

2026-09-26 16:47:30 +00:00

Long responses stream in bursts #59

krishsax added this to the (deleted) project

2026-09-26 19:06:50 +00:00

krishsax removed this from the (deleted) project

2026-09-26 19:07:10 +00:00

krishsax added this to the Agent project

2026-09-26 19:07:38 +00:00

Rows
Columns

Streaming slows down as responses get longer #24