HttpClient Fails to Get Access

Unity Engine
1

This may not be a Unity question, per se, but I’m seeing different behavior from Unity’s UnityWebRequest class than I am from .NET’s HttpClient, and I’m hoping someone can help me understand why.

This code runs fine:

using UnityEngine;
using UnityEngine.Networking;
using System.Collections;

public class UnityRead : MonoBehaviour
{
    void Start()
    {
        StartCoroutine(GetRequest("https://api.weather.gov"));
    }

    IEnumerator GetRequest(string uri)
    {
        using (UnityWebRequest webRequest = UnityWebRequest.Get(uri))
        {
            yield return webRequest.SendWebRequest();

            Debug.Log(webRequest.downloadHandler.text);
        }
    }
}

The server at https://api.weather.gov sends back this JSON:

"status": "OK"
}```

But when I try what I think is pretty much the same thing with .NET, like this:

```csharp
using System.Net.Http;
using UnityEngine;

public class NetRead : MonoBehaviour
{
    public string url = "https://api.weather.gov";

    void Start()
    {
        HttpClient client = new HttpClient();

        var getTask = client.GetAsync("https://api.weather.gov");
        getTask.Wait();

        var readTask = getTask.Result.Content.ReadAsStringAsync();
        readTask.Wait();

        Debug.Log(readTask.Result);
    }
}

the server answers me with this:

<TITLE>Access Denied</TITLE>
</HEAD><BODY>
<H1>Access Denied</H1>

You don't have permission to access "http://api.weather.gov/" on this server.<P>
Reference #18.4bdfda17.1656873718.52c0f19e
</BODY>
</HTML>```

I have the same problem when I run my .NET code in LINQPad (a lightweight IDE). Accessing the server from Chrome gets the "OK" message.

My code works fine when I access other servers, yet something about it fails to make api.weather.gov happy, while Unity's access works just fine. I've looked at the reference source, but it appears to be all just wrappers for native code, so I can't see what else Unity might be doing that my use of .NET's HttpClient class is leaving out. I'm not much of a Web maven, so I'm baffled.

Anyone know why Unity gets through while .NET doesn't?
2

Networking, UnityWebRequest, WWW, Postman, curl, WebAPI, etc:

And setting up a proxy can be very helpful too, in order to compare traffic:

3

Well, my problem’s kind of the reverse. It already is working in Unity. I’m trying to find out why it doesn’t work outside.

That might be helpful. My guess, after talking to my wife, is that the government expects some kind of certificate (or other authentication) with the request, to make sure they’re not under attack. Unity may well provide that, as does Chrome, I assume. Poor little me doesn’t have one, so my bare .NET code is rejected.

4

You need to add User-Agent header to request. Simplest is to add it as DefaultRequestHeaders.

Source: xml - Current Observation feed from weather.gov forbidden (403) - Stack Overflow

2 Likes
5

Thanks!

How did you manage to find that? I Googled for some time and got nothing remotely like it.

6

Interestingly, the fix at stackoverflow works for this URL:

https://w1.weather.gov/xml/current_obs/KIAD.xml

But not for this one:

https://api.weather.gov/stations/KIAD/observations/latest

I’m still far ahead of where I was yesterday, though, so thanks.

7

Ah, changing from POST to GET made it work. This is a bit afield from Unity, but might still be of use to someone finding their way here (and few things are more frustrating than a thread that starts with “Why doesn’t this work?” and then ends with “Okay, I figured it out.” and doesn’t show the answer).

void Main()
{
    HttpRequestMessage httpRequestMessage =
        new HttpRequestMessage(
                HttpMethod.Get,
                "https://api.weather.gov/stations/KCGS/observations/latest");

    httpRequestMessage.Headers.Add(
        "User-Agent",
        "MyApplication/v1.0 (http://foo.bar.baz; foo@bar.baz)");

    HttpClient httpClient = new HttpClient();

    Task<HttpResponseMessage> sendResponse = httpClient.SendAsync(httpRequestMessage);
    sendResponse.Wait();

    HttpContent httpContent = sendResponse.Result.Content;

    Task<string> readResponse = httpContent.ReadAsStringAsync();
    readResponse.Wait();

    Console.WriteLine(readResponse.Result);
}

Those Wait calls are probably not what you want in a Unity app, and I wouldn’t be surprised if someone with more experience in this area could improve upon the above in other ways. But it does work, so I’m posting it for whatever it’s worth.

2 Likes