Slow download using HttpClient

Unity Engine
1

I have been testing different methods to download files in Unity app. I tried using UnityWebRequest, HttpClient and using HttpWebRequest directly. These are the results for downloading a 27MB file on Android device connected to a 150Mbps WiFi network:

  • UnityWebRequest 2.1s
  • HttpWebRequest 3.5s
  • HttpClient 10s

The question is why is HttpClient so much slower than using HttpWebRequest directly? In the editor on Windows dowloads are much faster, but HttpClient is still twice as slow than HttpWebRequest.

I am using Unity 2019.1.14. This is the code that I use to get to the response input stream:

HttpClient:

// I only create HttpClient once and reuse it for all requests
var httpClient = new HttpClient { Timeout = TimeSpan.FromMilliseconds(15000) };
var response = await httpClient.GetAsync(url, , HttpCompletionOption.ResponseHeadersRead).ConfigureAwait(false);

var readStream = await response.Content.ReadAsStreamAsync().ConfigureAwait(false);

HttpWebRequest:

var request = (HttpWebRequest)WebRequest.Create(url);
request.Timeout = 15000;
request.ReadWriteTimeout = 5000;

var response = (HttpWebResponse)request.GetResponse();

var readStream = response.GetResponseStream();

Once I have the stream I process it in a while loop and this is the part that is slower for HttpClient:

var buffer = new byte[65536];
while ((read = stream.Read(buffer, 0, buffer.Length)) > 0)
{
    // Write buffer to file output stream
}

Any ideas what could be the problem here?

If anyone is interested why would we want to use anything else except UnityWebRequest, the reason is that with UnityWebRequest the whole content will get loaded into memory at once and you have to process it on main thread which will cause your game to stutter. Using these other methods you can process the data as it comes and on a separate thread so your game can run smoothly.

2

Are you sure about this? I thought you could implement your own downloader mechanism:

https://docs.unity3d.com/Manual/UnityWebRequest-CreatingDownloadHandlers.html

Caveat: I haven’t tried it, but I know it only took like 4mb to open a giant ~100mb asset bundle and read a single specific file out of it.

3

It’s really difficult to believe that any sort of client optimization issue could be costing you 7-8 seconds on a 2-3 sec web download. I think it’s much more likely that the difference somehow involves a problem with your testing procedure and/or some difference in the network. Maybe something involving caching (especially if the first test was the slow one), or maybe some throttling by your ISP? If you only tested each method once, it could even just be a fluke of how congested the network was at that exact moment.

I suggest you make sure you are doing multiple tests with each method, and in a variable order.

1 Like
4

@Kurt-Dekker you are right that we can use custom DownloadHandler to process data as it comes but the processing will still happen on main thread that will affect the game.

@Antistone I repeated the tests many, many times in different order at my office network and at my home network (The numbers I gave are the average values on my home network). I was adding additional logging and trying to write things a bit differently here and there to see if anything will affect the speed. I even get the same slow down when testing on PC in Unity Editor.

5

It turned out I had a bug in the initialization and I actually wasn’t reusing HttpClient but creating a new instance every time I run a test. I fixed that and timing between HttpWebRequest and HttpClient in the Editor is now about the same, but on the device HttpClient is still slower. These are the new timings:

  • UnityWebRequest 2.1s
  • HttpWebRequest 3.5s
  • HttpClient 4s