TLS errors diagnostic
2021-11-19So your .NET application is trying to call an https endpoint and fails with a message similar to the following?
System.Net.Http.HttpRequestException:
An error occurred while sending the request.
---> System.Net.WebException:
The request was aborted: Could not create SSL/TLS secure channel.
Or maybe like this one?
System.Net.Http.HttpRequestException:
An error occurred while sending the request.
---> System.Net.WebException:
The underlying connection was closed: Could not establish trust relationship for the SSL/TLS secure channel.
---> System.Security.Authentication.AuthenticationException:
The remote certificate is invalid according to the validation procedure.
Let's see what we can do...
Note: I will assume we are on Windows and using the native TLS implementation (Schannel)
Table of content
Possible causes
Among the most likely causes for these errors:
- Mismatched TLS version between client and server.
- No common cipher suite between client and server.
- The server certificate is considered as invalid by the client.
- It might have expired (or its validity could be in the future, or the client and server clocks might be desynchronized)
- It might not be signed by a certificate authority the client trusts (or the signature could be invalid)
- It might not be valid for the DNS name we are trying to access.
- It might have been revoked (though I don't think the revocation status is checked unless explicitly configured to do so by the client)
Handshake failure
The first two problems will trigger a TLS handshake failure (error code 40).
No more information will be available for the simple reason that TLS does not provide any: if the server cannot communicate with the client, it will send a TLS alert and terminate the connection.
Bad certificate
For the case where the connection fails because of an invalid certificate (error code 42), the TLS alert will mention it, but it will be encrypted.
You will need the decrypted conversation to see the actual error (or rely on information provided by your application).
Diagnosing with .NET network tracing
Note: The following is for a .NET Framework application. It's probably not directly applicable for a .NET Core application...
This method will get us all the information Schannel is able to report, without altering the system configuration or requiring admin rights.
It is very helpful to diagnose bad certificates, but will not be sufficient to diagnose the cause of a handshake failure (because TLS does not report the cause of such errors).
.NET Framework comes with built-in network tracing, but it's off by default.
You will need to edit your App.config to enable it. Report to the Microsoft documentation for details.
If the problem was caused by a bad certificate, the resulting log should give you enough information to diagnose the cause of the error.
You might also get a simple IllegalMessage error that could apply to multiple situations. In this case, you will have to make a network capture (see the next section about that).
Here is an example trace log where the server certificate was misconfigured (the DNS name in the certificate and the actual server host name were different):
(sensitive data removed)
System.Net Information: 0 : [33444] InitializeSecurityContext(In-Buffers count=2, Out-Buffer length=0, returned code=OK).
System.Net Information: 0 : [33444] Remote certificate: [Version]
V3
[Subject]
CN=...
Simple Name: ...
DNS Name: ...
[Issuer]
CN=R3, O=Let's Encrypt, C=US
Simple Name: R3
DNS Name: R3
[Serial Number]
...
[Not Before]
2021-11-08 ...
[Not After]
2022-02-06 ...
[Thumbprint]
...
[Signature Algorithm]
sha256RSA(1.2.840.113549.1.1.11)
[Public Key]
Algorithm: RSA
Length: 2048
Key Blob: ...
System.Net Information: 0 : [33444] SecureChannel#41622463 - Remote certificate has errors:
System.Net Information: 0 : [33444] SecureChannel#41622463 - Certificate name mismatch.
System.Net Information: 0 : [33444] SecureChannel#41622463 - Remote certificate was verified as invalid by the user.
System.Net Information: 0 : [33444] ApplyControlToken() returned OK.
System.Net Information: 0 : [33444] InitializeSecurityContext(credential = System.Net.SafeFreeCredential_SECURITY, context = 51ed420:51efc08, targetName = [...], inFlags = ReplayDetect, SequenceDetect, Confidentiality, AllocateMemory, InitManualCredValidation)
System.Net Information: 0 : [33444] InitializeSecurityContext(In-Buffer length=0, Out-Buffer length=31, returned code=OK).
System.Net Error: 0 : [23832] Exception in HttpWebRequest#58870012:: - The underlying connection was closed: Could not establish trust relationship for the SSL/TLS secure channel..
Diagnosing with network captures
The .NET level tracing is sufficient for bad certificates, but diagnosing TLS handshake failures is harder: the connection is closed without details about the failure cause.
The only way I found for these cases is to capture actual network traffic and manually analyze it.
1. Capture traffic
The first step is to capture the relevant TLS packets.
To do this, use Wireshark/tshark (the CLI Wireshark).
Or use the built-in Windows capture tool (requires admin privileges, like Wireshark):
netsh trace start capture=yes report=disabled tracefile=trace.etl
netsh trace stop
I will not detail the procedure here. Just make sure you are capturing traffic on the correct interface (or for all interfaces if you are not sure)...
The netsh command outputs an ETL file that needs to be converted to a pcapng so we can analyze it with Wireshark/tshark.
Use this tool for conversion: github.com/microsoft/etl2pcapng
etl2pcapng.exe trace.etl trace.pcapng
2. Analyze TLS errors
It's possible to use Wireshark, but since I want to try and automate this process as much as possible, I will be using tshark with specially tailored filters.
The steps are similar and filters are the same for Wireshark.
Here is the official tshark manual: www.wireshark.org/docs/man-pages/tshark.html
tshark options
-2- Perform a two-pass analysis. This causes tshark to buffer output until the entire first pass is done, but allows it to fill in fields that require future knowledge, such as 'response in frame #' fields. Also permits reassembly frame dependencies to be calculated correctly.
-o "tcp.reassemble_out_of_order:true"- Often necessary for tshark to parse HTTP2/TLS streams properly (Without it, the certificate handshake is not decoded) You might need to set this option for Wireshark analysis too (in the TCP preferences, this is the "Reassemble out-of-order segments" checkbox)
-o "gui.column.format:..."- Only way I found to add the stream number in the output data without losing the default useful information.
tshark option to display only some fields (-T fields and -e ...) cannot output human readable info...
- Only way I found to add the stream number in the output data without losing the default useful information.
-O tls- Output all information available about the decoded packets for the specified protocol (tls here)
-Y- Display filter. (specifying -Y instead of simply putting the filter at the end allows to put it anywhere in the command)
-r- Read a pcapng file as input.
Wireshark/tshark filters
tls.record.content_type eq 21- Only display TLS alert packets
tls.record.content_type in {21 22}- Display both TLS alerts and TLS handshake packets (which contain all the crypto negotiation and certificate/key exchanges)
tcp.stream eq 14- Filter for a specific TCP stream
Ready-made commands
The following commands can be used as-is (just replace mycapture.pcapng with your file)
Note the commands are escaped for the classic Windows CLI.
With Powershell, it's possible to use single quotes, so the escaping becomes something like the following:
./tshark --color -2 -o 'tcp.reassemble_out_of_order:true' -o 'gui.column.format:"Time","%Yt","Stream","%Cus:tcp.stream","Source","%s","Destination","%d","Protocol","%p","Info","%i"' -Y 'tls.record.content_type in {21 22} && tcp.stream eq 14' -r mycapture.pcapng
Display all the TLS alerts (content type 21) in the pcap.
The second column is the TCP stream number.
"C:\Program Files\Wireshark\tshark.exe" --color -2 -o "tcp.reassemble_out_of_order:true" -o "gui.column.format:\"Time\",\"%Yt\",\"Stream\",\"%Cus:tcp.stream\",\"Source\",\"%s\",\"Destination\",\"%d\",\"Protocol\",\"%p\",\"Info\",\"%i\"" -Y "tls.record.content_type eq 21" -r mycapture.pcapng
In this capture, the two TLS alerts happening at the same time are actually the same one, captured on two different interfaces, one physical, one from a virtual machine.
Let's analyze the first handshake failure.
The following will display TLS alerts and TLS handshake packets (content type 21 and 22) for stream 14.
(With Wireshark, you would instead right click the packet and select "Follow > TCP Stream".)
"C:\Program Files\Wireshark\tshark.exe" --color -2 -o "tcp.reassemble_out_of_order:true" -o "gui.column.format:\"Time\",\"%Yt\",\"Stream\",\"%Cus:tcp.stream\",\"Source\",\"%s\",\"Destination\",\"%d\",\"Protocol\",\"%p\",\"Info\",\"%i\"" -Y "tls.record.content_type in {21 22} && tcp.stream eq 14" -r mycapture.pcapng
Add -O tls to output the whole decoded packets for the TLS protocol, with the selected packets.
In Wireshark, you would instead display the details of theClient Hello packet.
"C:\Program Files\Wireshark\tshark.exe" --color -2 -o "tcp.reassemble_out_of_order:true" -o "gui.column.format:\"Time\",\"%Yt\",\"Stream\",\"%Cus:tcp.stream\",\"Source\",\"%s\",\"Destination\",\"%d\",\"Protocol\",\"%p\",\"Info\",\"%i\"" -Y "tls.record.content_type in {21 22} && tcp.stream eq 14" -r mycapture.pcapng -O tls
The result of this command will probably be a lot of text. You can add > out.log to output all the data to a new out.log file.
Here is the complete and unaltered output of the previous command:
Frame 1426: 228 bytes on wire (1824 bits), 228 bytes captured (1824 bits) on interface \Device\NPF_{465FCAB1-67D3-4DD1-BE13-F37A177C1187}, id 12
Ethernet II, Src: Microsof_01:3e:00 (00:15:5d:01:3e:00), Dst: Microsof_64:a0:ad (00:15:5d:64:a0:ad)
Internet Protocol Version 4, Src: 172.21.226.129, Dst: 185.245.136.237
Transmission Control Protocol, Src Port: 50159, Dst Port: 443, Seq: 1, Ack: 1, Len: 174
Transport Layer Security
TLSv1.2 Record Layer: Handshake Protocol: Client Hello
Content Type: Handshake (22)
Version: TLS 1.2 (0x0303)
Length: 169
Handshake Protocol: Client Hello
Handshake Type: Client Hello (1)
Length: 165
Version: TLS 1.2 (0x0303)
Random: 6197c7e2272a02f0845d1d4d00b4ed06a65d500d35a6b3bd9bd39bddc03d1edc
GMT Unix Time: Nov 19, 2021 16:50:58.000000000 Romance Standard Time
Random Bytes: 272a02f0845d1d4d00b4ed06a65d500d35a6b3bd9bd39bddc03d1edc
Session ID Length: 0
Cipher Suites Length: 42
Cipher Suites (21 suites)
Cipher Suite: TLS_RSA_WITH_AES_128_CBC_SHA256 (0x003c)
Cipher Suite: TLS_RSA_WITH_AES_128_CBC_SHA (0x002f)
Cipher Suite: TLS_RSA_WITH_AES_256_CBC_SHA256 (0x003d)
Cipher Suite: TLS_RSA_WITH_AES_256_CBC_SHA (0x0035)
Cipher Suite: TLS_RSA_WITH_RC4_128_SHA (0x0005)
Cipher Suite: TLS_RSA_WITH_3DES_EDE_CBC_SHA (0x000a)
Cipher Suite: TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256 (0xc027)
Cipher Suite: TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA (0xc013)
Cipher Suite: TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA (0xc014)
Cipher Suite: TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256 (0xc02b)
Cipher Suite: TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256 (0xc023)
Cipher Suite: TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384 (0xc02c)
Cipher Suite: TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA384 (0xc024)
Cipher Suite: TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA (0xc009)
Cipher Suite: TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA (0xc00a)
Cipher Suite: TLS_DHE_DSS_WITH_AES_128_CBC_SHA256 (0x0040)
Cipher Suite: TLS_DHE_DSS_WITH_AES_128_CBC_SHA (0x0032)
Cipher Suite: TLS_DHE_DSS_WITH_AES_256_CBC_SHA256 (0x006a)
Cipher Suite: TLS_DHE_DSS_WITH_AES_256_CBC_SHA (0x0038)
Cipher Suite: TLS_DHE_DSS_WITH_3DES_EDE_CBC_SHA (0x0013)
Cipher Suite: TLS_RSA_WITH_RC4_128_MD5 (0x0004)
Compression Methods Length: 1
Compression Methods (1 method)
Compression Method: null (0)
Extensions Length: 82
Extension: renegotiation_info (len=1)
Type: renegotiation_info (65281)
Length: 1
Renegotiation Info extension
Renegotiation info extension length: 0
Extension: server_name (len=28)
Type: server_name (0)
Length: 28
Server Name Indication extension
Server Name list length: 26
Server Name Type: host_name (0)
Server Name length: 23
Server Name: snc18pp.snc.oodrive.com
Extension: status_request (len=5)
Type: status_request (5)
Length: 5
Certificate Status Type: OCSP (1)
Responder ID list Length: 0
Request Extensions Length: 0
Extension: supported_groups (len=6)
Type: supported_groups (10)
Length: 6
Supported Groups List Length: 4
Supported Groups (2 groups)
Supported Group: secp256r1 (0x0017)
Supported Group: secp384r1 (0x0018)
Extension: ec_point_formats (len=2)
Type: ec_point_formats (11)
Length: 2
EC point formats Length: 1
Elliptic curves point formats (1)
EC point format: uncompressed (0)
Extension: signature_algorithms (len=16)
Type: signature_algorithms (13)
Length: 16
Signature Hash Algorithms Length: 14
Signature Hash Algorithms (7 algorithms)
Signature Algorithm: rsa_pkcs1_sha256 (0x0401)
Signature Hash Algorithm Hash: SHA256 (4)
Signature Hash Algorithm Signature: RSA (1)
Signature Algorithm: rsa_pkcs1_sha384 (0x0501)
Signature Hash Algorithm Hash: SHA384 (5)
Signature Hash Algorithm Signature: RSA (1)
Signature Algorithm: rsa_pkcs1_sha1 (0x0201)
Signature Hash Algorithm Hash: SHA1 (2)
Signature Hash Algorithm Signature: RSA (1)
Signature Algorithm: ecdsa_secp256r1_sha256 (0x0403)
Signature Hash Algorithm Hash: SHA256 (4)
Signature Hash Algorithm Signature: ECDSA (3)
Signature Algorithm: ecdsa_secp384r1_sha384 (0x0503)
Signature Hash Algorithm Hash: SHA384 (5)
Signature Hash Algorithm Signature: ECDSA (3)
Signature Algorithm: ecdsa_sha1 (0x0203)
Signature Hash Algorithm Hash: SHA1 (2)
Signature Hash Algorithm Signature: ECDSA (3)
Signature Algorithm: SHA1 DSA (0x0202)
Signature Hash Algorithm Hash: SHA1 (2)
Signature Hash Algorithm Signature: DSA (2)
Frame 1427: 61 bytes on wire (488 bits), 61 bytes captured (488 bits) on interface \Device\NPF_{465FCAB1-67D3-4DD1-BE13-F37A177C1187}, id 12
Ethernet II, Src: Microsof_64:a0:ad (00:15:5d:64:a0:ad), Dst: Microsof_01:3e:00 (00:15:5d:01:3e:00)
Internet Protocol Version 4, Src: 185.245.136.237, Dst: 172.21.226.129
Transmission Control Protocol, Src Port: 443, Dst Port: 50159, Seq: 1, Ack: 175, Len: 7
Transport Layer Security
TLSv1.2 Record Layer: Alert (Level: Fatal, Description: Handshake Failure)
Content Type: Alert (21)
Version: TLS 1.2 (0x0303)
Length: 2
Alert Message
Level: Fatal (2)
Description: Handshake Failure (40)
Manual analysis
Ok, so we have the complete information our client application sent to the server, just before the server closed the connection.
What do we do with this?
We only have information about the client, but what about the server?
You might know the TLS server configuration, but let's make sure.
SSL Labs
SSL Labs will analyse the server and report what TLS versions and cipher suites it supports.
In our case, we see that the server only supports TLS 1.2.
Since our client sent a TLS 1.2 client hello, the problem is elsewhere.
SSL Labs also reports the cipher suites supported:
TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (0xc030)TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305_SHA256 (0xcca8)TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256 (0xc02f)
And is friendly enough to simulate connections with various clients.
Among them we can see several failures.
This is where the dots connect: it turns out our client app is running on Windows 7, and none of the Schannel cipher suites enabled on this OS are compatible with the cipher suites advertised by the server!
The sever was hardened to support only strong cipher suites (ciphersuite.info is a good source), and inadvertently dropped Windows 7 compatibility in the process.
Nmap
SSL Labs is more friendly, but nmap can also be used to do the same thing, and it's faster:
$ nmap --script ssl-enum-ciphers -p 443 snc18pp.snc.oodrive.com
Starting Nmap 7.80 ( https://nmap.org ) at 2021-11-19 17:06 CET
Nmap scan report for snc18pp.snc.oodrive.com (185.245.136.237)
Host is up (0.0031s latency).
PORT STATE SERVICE
443/tcp open https
| ssl-enum-ciphers:
| TLSv1.2:
| ciphers:
| TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (secp384r1) - A
| TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305_SHA256 (secp384r1) - A
| TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256 (secp384r1) - A
| compressors:
| NULL
| cipher preference: server
|_ least strength: A
Nmap done: 1 IP address (1 host up) scanned in 1.68 seconds
You will then have to manually try to match a cipher suite between the server and the client...
Curl
If you want to do some quick testing with another client application, curl can be helpful.
Be careful though: on Windows, depending on how it's compiled, curl might use openssl instead of Schannel.
It means that curl might succeed while applications using the OS TLS stack will not.
Some versions of curl are compiled to support mutiple TLS implementations.
In this case, you can switch between them by setting an environment variable.
For example to use Schannel:
set CURL_SSL_BACKEND=schannel
curl -v https://snc18pp.snc.oodrive.com
To use a specific TLS version, use --tls-max 1.0 (the previous works with Schannel, while --tlsv1.0 only works with openssl)
With some TLS implementations, you can restrict which cipher suites to use with --ciphers. More info here: curl.se/docs/ssl-ciphers.html
Conclusion
I hope this was helpful.
Since capturing network packets requires admin privileges, specific tools knowledge, and might be a non-starter on production servers, I looked for a way to diagnose TLS handshake failures without resorting to them, but beside guessing, it still seems this is the only method...
There is another way to log Schannel events I didn't talk about, because I didn't find it very useful compared to the two methods I documented above.
If you are not able to add traces to your app though, it might interest you: kevinjustin.com/blog/2017/11/08/schannel-event-logging
If you have another solution, please share!
The comment is shown highlighted below in context.
JavaScript is required to see the comments. Sorry...