TlsException: Handshake failed - error code: UNITYTLS_INTERNAL_ERROR when trying to authenticate websockets

So I have a custom multiplayer system using sslstream sockets on the server trying to connect to a websocket from a simple javascript plugin on the client (using a normal socket server side that upgrades to a websocket because of cross platform stuff) it used to work but lately I’ve been getting this error everytime I call the SslStream.AuthenticateAsServer() function:

TlsException: Handshake failed - error code: UNITYTLS_INTERNAL_ERROR

The authentication works fine when connecting to a standalone client that’s also using a normal sslstream instead of a websocket so I suspect its some configuration setting I need to figure out. Any ideas?

has the certificate changed recently?

c# - Unity TlsException: Handshake failed UNITYTLS_X509VERIFY_FLAG_NOT_TRUSTED - Stack Overflow

It’s with let’s encrypt so it autorenews periodically but the path is the same, also it should be fine because like I mentioned I can connect with a regular sslstream and authenticate it, just not with a JavaScript websocket. Also that’s a different error I don’t get the UNITYTLS_X509VERIFY_FLAG_NOT_TRUSTED just the UNITYTLS_INTERNAL_ERROR

is that because you dont inspect it further?

this is the full error that shows in the console

TlsException: Handshake failed - error code: UNITYTLS_INTERNAL_ERROR
Mono.Unity.Debug.CheckAndThrow (Mono.Unity.UnityTls+unitytls_errorstate errorState, System.String context, Mono.Security.Interface.AlertDescription defaultAlert) (at <e243450672844c6f9f403082ca9f1be7>:0)
Mono.Unity.UnityTlsContext.ProcessHandshake () (at <e243450672844c6f9f403082ca9f1be7>:0)
Mono.Net.Security.MobileAuthenticatedStream.ProcessHandshake (Mono.Net.Security.AsyncOperationStatus status, System.Boolean renegotiate) (at <e243450672844c6f9f403082ca9f1be7>:0)
(wrapper remoting-invoke-with-check) Mono.Net.Security.MobileAuthenticatedStream.ProcessHandshake(Mono.Net.Security.AsyncOperationStatus,bool)
Mono.Net.Security.AsyncHandshakeRequest.Run (Mono.Net.Security.AsyncOperationStatus status) (at <e243450672844c6f9f403082ca9f1be7>:0)
Mono.Net.Security.AsyncProtocolRequest.ProcessOperation (System.Threading.CancellationToken cancellationToken) (at <e243450672844c6f9f403082ca9f1be7>:0)
Rethrow as AuthenticationException: Authentication failed, see inner exception.
Mono.Net.Security.MobileAuthenticatedStream.ProcessAuthentication (System.Boolean runSynchronously, Mono.Net.Security.MonoSslAuthenticationOptions options, System.Threading.CancellationToken cancellationToken) (at <e243450672844c6f9f403082ca9f1be7>:0)
Networking.servePlayer (System.Net.Security.SslStream stream, System.Int32 userId) (at Assets/Scripts/Networking/Networking.cs:164)
System.Runtime.CompilerServices.AsyncMethodBuilderCore+<>c.<ThrowAsync>b__7_0 (System.Object state) (at <ed969b0e627d471da4848289f9c322df>:0)
UnityEngine.UnitySynchronizationContext+WorkRequest.Invoke () (at <44f3679c53d1477a9c6e72f269e3a3a9>:0)
UnityEngine.UnitySynchronizationContext.Exec () (at <44f3679c53d1477a9c6e72f269e3a3a9>:0)
UnityEngine.UnitySynchronizationContext.ExecuteTasks () (at <44f3679c53d1477a9c6e72f269e3a3a9>:0)

If you have a way to inspect further beyond the console i would love to hear about it

any ideas? still stuck, considering trying to figure out how to submit a bug report to unity.

well submitting a bug report is easy, its a menu item (help->report a bug), however, it will zip up everything, so it is best to make a new project with a minimal case to demonstrate

update: I’m a dum dum certs don’t always work for subdomains but apparently do in the standalone build?
I had been using gameserver.[actualdomain].[tld] so I could setup an actual website at [actualdomain].[tld] and it turns out that’s what was causing the auth to fail, setting it back to just use [actualdomain].[tld] fixed it so ill just need to figure out how to setup the cert to specifically apply to the subdomain.

I thought certs were pretty anal, and needed to match server name, subdomain or not , getting a generic global cert is pretty expensive