I've been writing a Windows console application using Casablanca, but I've been having performance issues on transfer rate. Metrics show that the console application is transferring a byte stream at roughly 6-8 Mbytes/sec.
To diagnose this, we looked at the application's file read speeds. It was getting anywhere from 20 Mbytes/sec to 100 Mbytes/sec. The variation depends on the size of the request. Read lots of small chunks, and throughput goes down. On average, file reads were at 80 Mbytes/sec.
We then tested Casablanca for local host to local host transfer with large file reads (to eliminate the variation.) We got transfer rates of about 60 Mbytes/sec. But when we tested the application between a virtual machine on the network and a physical machine, the transfer rate was at 6-8 Mbytes/sec.
Since the drop in performance occurred going from local host loop to real network, we then used curl to test a 1Gig download on the actual network. We got transfer rates of around 60 Mbytes/sec.
Here's what we think is happening:
- Disk read isn't the bottleneck. Speed here is around 80 Mb/s or higher.
- Local loop isn't the bottleneck. Speed here is around 60 Mb/s or higher.
- Network isn't the bottleneck. Speed here between IIS and curl is around 60 Mb/s or higher.
-
Casablanca seems to be the bottleneck. Speed here goes from 80 Mb/s to 8 Mb/s.
We looked at the network load monitor: Between IIS and curl, the transfer had a ramp up, a flat plateau, and then a ramp down. As expected. Between Casablanca and curl, it was jagged spikes all the way, and it never really reach the higher transfer rates.
As first I thought, maybe I was using the streaming wrongly. The console application uses a producer-consumer buffer and ties an iStream and oStream to it. We do a write to it after a disk read. Debug logged output shows that the application quickly fills the REST buffer and is done with the disk read. The rest of the time is spent by Casablanca pushing out the buffer to network.
But when we tested with a code change (changed the code to not stream, but to read in the entire request into memory and then send it all at once in the reply body,) the change was negligible. Network monitoring showed Casablanca still dumped the 1Gig file in little bursts, constant spikes.
I'm wondering what we're doing wrong here. Can anyone offer any insight?