Yes, that should work, the http_client is pushing data through as soon as it arrives, but if the system is throttling or buffering data somewhere else in the system, we don't push it through until it's given. Our next release will allow you to set the size of the chunks the client asks for, too.
The callback you're looking for is what you get if instead of calling .get(), you call .then(). Asynchronous processing in a loop is hard, it requires that you radically rewrite your code. For an example, you may want to look at the implementation of read_line(), where we loop reading from a stream buffer until a newline character is found. Nasty-looking code, but completely unblocking.
Niklas
The callback you're looking for is what you get if instead of calling .get(), you call .then(). Asynchronous processing in a loop is hard, it requires that you radically rewrite your code. For an example, you may want to look at the implementation of read_line(), where we loop reading from a stream buffer until a newline character is found. Nasty-looking code, but completely unblocking.
Niklas