When accessing a gRPC streaming interface (converted to HTTP) with cURL before, a very strange phenomenon occurred.
- when directly
cuRL, always the full content is displayed and the connection does not break, because it is a streaming interface, which is exactly the expected behavior.
- And when
cURL | ...or
cURL > file, the full content is never available, and it doesn’t work no matter how long you wait.
Why is this? This problem has been bothering me for a long time. Until yesterday when I suddenly had an idea why.
Probably because of cURL’s buffer.
So I read the manual of cURL, and there is really one buffer-related issue:
Then I tried to turn it on, and the problem was solved.
However, there is still one question, why does the direct
cURL always show the full content? Shouldn’t it also buffer up? No answer found. Blind guess is that it’s the version.
I wrote a small piece of code to try to reproduce the problem.
Note: The reason for adding
Transfer-Encoding is that the go language chunked transfers by default, and I don’t know how to simply turn it off.
Since the buffer size of cURL is 16KB, I constructed my code with a little more than 16KB of data.
Now to try to request this interface.
Sure enough, it stops here, and the 456789 that follows is gone. As long as the code sleeps long enough, it will keep waiting here.
But the weird thing is that I can’t reproduce the above problem completely, whether I redirect to the file or pipe it to another command (I’m too lazy to write the grpc example).
Also, cURL can specify how long to stop the download if the speed continues to fall below a certain value (returning error code 28: Operation timeout):.
The service I mentioned earlier is a streaming subscription service that starts with a full subscription and then subscribes to incremental data (less), so the speed generally goes to zero after the subscription, which for me made these parameters work well. I ended up with the following command.
As for the very first question: output to terminal is no problem, output to file or pipe is a problem, and I still don’t have an answer.