Why Certificates and What is Cert-Pinning?

In the internet age, it has become mandatory for all organisations and services which are hosted on the internet to rely on certificate based authentication and authorisation in order to protect the confidentiality and integrity of their data accessible and traversing over the internet. Certificates help prove your identity over the internet when communicating with parties/services which you have not interacted with before. As organisations and service providers started heavily relying on certificates for their security needs, they wanted to be sure that there was no chance of data leakage through techniques such as man-in-the-middle attacks. This led to the rise of adoption of the concepts like trusted certificate stores and certificate pinning.

Certificate pinning is a security method which protects the service providers against man-in-the-middle attacks. The server/host is associated with a specific certificate or public key and the applications which reach out to the particular server expect that particular certificate or public key when trying to initiate a connection.  When initiating a request, the application verifies the server's certificate against the pinned one and proceeds only if it matches. If there is a slight mismatch, the process/application rejects the entire transaction as the application cannot confirm if it is connecting to the server/host it is supposed to connect to.

Netskope Client and Cert-Pinning

The Netskope Client plays a crucial role in providing security services like Data Loss Prevention (DLP) through deep packet inspection. However, the inherent nature of SSL/TLS encryption presents a challenge to this inspection process. When a client establishes a secure connection with a server using SSL/TLS, the data exchanged between them is encrypted, rendering it inaccessible to external monitoring, including the Netskope Client. To overcome this limitation and enable packet inspection of encrypted traffic, the Netskope Client operates as a trusted intermediary during the initial handshake of an SSL/TLS connection. When a client initiates a secure transaction with a server, the Netskope Client intercepts this connection establishment phase. In this interception, the Netskope Client introduces its own certificate into the communication flow, positioning itself between the client and the server.

This "man-in-the-middle" approach allows the Netskope Client to effectively decrypt the SSL/TLS traffic. By presenting its certificate, the Netskope Client can establish a secure connection with both the client (acting as the server from the client's perspective) and the server (acting as the client from the server's perspective). This dual secure connection enables the Netskope Client to decrypt the traffic flowing from the client, inspect it for policy violations (such as DLP rules), and then re-encrypt it before forwarding it to the intended server. The reverse process occurs for traffic flowing from the server back to the client.

However, the introduction of the Netskope certificate creates a discrepancy from the client's perspective. Typically, when a client connects to a secure website or application, it expects to receive the server's specific SSL/TLS certificate. This certificate is often "pinned" or validated by the client or the application to ensure the authenticity and integrity of the server it is communicating with. When the Netskope Client intercepts the connection and presents its own certificate instead of the expected server certificate, the client or the application recognizes this as an anomaly. The received Netskope certificate does not match the server's pinned certificate, leading the client to distrust the connection. Consequently, the client, adhering to security protocols designed to prevent man-in-the-middle attacks, will typically terminate the connection, resulting in a certificate error.

Why does this not happen with every application?

While the certificate error previously discussed might not manifest universally across all applications, this variability arises from differing approaches applications take in verifying the security certificates involved in their transactions. Some applications possess more lenient certificate checking mechanisms, potentially overlooking specific details during the validation process. Conversely, other applications delegate the responsibility of certificate trust entirely to the underlying operating system. These operating system-reliant applications leverage the system's built-in certificate store as their source of truth for trusted certificates. When an application encounters a certificate during a secure communication attempt, it queries the operating system's certificate store to ascertain the certificate's validity and trustworthiness.

This is where the Netskope client plays a crucial role in ensuring seamless and secure operation. Upon installation on a machine, the Netskope client proactively installs its own certificate within the operating system's central certificate store. This action establishes the Netskope certificate as a trusted entity at the operating system level. Consequently, when an application that relies on the operating system's certificate store encounters a transaction that involves the Netskope client's certificate (for instance, during traffic inspection or redirection), it automatically finds the Netskope certificate already present and validated within its trusted store. This pre-existing trust relationship eliminates the need for the application to perform independent or more stringent certificate checks. As a result, the application implicitly trusts the Netskope certificate and proceeds with the transaction without raising any certificate-related errors or interruptions. This mechanism ensures a smooth user experience while allowing the Netskope client to perform its security functions transparently.

Configuring Applications and Tools to work with Netskope Inspection

As discussed earlier, Netskope's inline deployment actively intercepts SSL/TLS encrypted traffic, frequently employing its own Certificate Authority (CA) for this purpose. Certain Command Line Interface (CLI) tools, particularly those prevalent in programming languages such as Python, often rely on their own dedicated certificate bundles for verifying secure connections. Consequently, these tools might not inherently trust the Netskope CA, leading to potential connection failures. To address this trust discrepancy and ensure seamless operation, it is necessary to explicitly configure these tools to recognize and trust the Netskope CA. This configuration typically involves setting specific environment variables within the operating system or the tool's execution environment, where these variables point to the file containing the Netskope CA certificate. By establishing this trust, the CLI tools can successfully establish secure connections even when Netskope's SSL interception is active. The precise configuration steps and environment variable names can vary depending on the specific CLI tool and the underlying library it utilizes, so consulting the relevant tool and library documentation is crucial for accurate implementation.

This document talks in details about the process mentioned above: Configuring CLI-based Tools and Development Frameworks to work with Netskope SSL Interception 

Bypassing Traffic - Creating Certificate Pinned Applications

Bypassing Netskope Points of Presence (POPs) can be employed as an alternative method to resolve certificate-related errors. This involves configuring applications to route their traffic directly to the internet, effectively circumventing the Netskope infrastructure. Establishing a direct connection in this manner often eliminates immediate certificate errors. However, this approach comes at a significant cost to security. By bypassing Netskope POPs, organizations lose crucial security visibility into their application traffic. Consequently, vital Netskope security services such as Data Loss Prevention (DLP) and Threat Protection are unable to inspect the traffic, leaving the organisation vulnerable to potential threats and data loss.

You can create a bypass by navigating to the following section in your Netskope Tenant

• Settings → Security Cloud Platform → App Definition → Certificate Pinned Apps (Tab)

You specify the exact process from which the traffic needs to be bypassed. You can use the Client logs to identify the process and the destination IP Address when you recreate the issue. You can recreate the issue and note down the local system time when the certificate error occurred and save the Netskope Client logs. Unzip and open the log bundle and open the ‘nsdebuglog.log’ file and search for the date and timestamp when you had recreated the issue. You can look for the process and the IP Address/Domain Name and note it down to add it in the Netskope Tenant. Adding an example in the below screenshot, where you can correlate the commands run in the Mac terminal which have failed due to the certificate error and the Netskope log bundle recording the possible traffic and the process information.

You can create a Certificate Pinned Application definition in the tenant based on the process name using the platform (Operating System) details and the exact process name or regular expression.

Bypassing Traffic - Steering Configuration Exceptions

The next step is to exclude the particular process in the respective Steering Configuration in Netskope. You can navigate to create an exception in the Steering Configuration section by

• Settings → Security Cloud Platform → Steering Configuration → Navigate to the Group → Exceptions (Tab) → New Exception → Certificate Pinned Application

Click on the respective Steering Configuration group which requires the exception. For example, only the folks from the Engineering department might be using python and the exception may be required only for them. So instead of having a single steering configuration, you can create a dedicated configuration for the ‘Engineering’ department and add the exception for this group alone so that this does not apply to other departments like Finance, HR, etc. You will have to keep in mind that you can only allow bypassing of certain domains in the steering configuration. For example, if you want to bypass traffic from the ‘Python’ process only if it reaches out to a domain ‘pythoni.]org’, you can create an exception in such a way that the traffic from the process ‘Python’ is bypassed only if the destination domain is ‘pythons.]org’, all other traffic from the process ‘Python’ will still be steered through the Netskope POP.

Understanding ‘Advanced Options’ in the Exception

In the above screenshot, we could see 2 advanced options while creating the exception. They both have a significant use case to solve while creating and handling exceptions. With the first option ‘Devices matching specific Device Classification’, the admin can choose to apply the steering configuration bypass only for devices which match a specific Steering Configuration. For example, the bypass may apply to only corporate managed devices and similar traffic from an unmanaged device would still be tunneled through the Netskope POP. 

The other option is ‘Tunnel Mode’ which can be used to increase the security posture of the bypassed traffic. If the traffic is bypassed without ‘Tunnel Mode’, the traffic directly leaves to the destination from the source without touching the Netskope POP. So the request IP seen at the destination host/application would be that of the Internet Service Provider of the client. But if ‘Tunnel Mode’ is enabled, the traffic will be steered to the Netskope POP but the traffic will leave the POP uninspected. Here Netskope acts as a proxy which does not perform packet inspection for DLP or Threat Protection. So here the request IP seen at the destination host/application would be that of a Netskope POP belonging to the Netskope IP Address range. With this feature, you could Whitelist/Allow list the Netskope Egress IP ranges at the destination and be sure that access to the resource is granted only if the request IP is from a Netskope IP Address range for enhanced security while not performing any inspection activities on the traffic leading to certificate issues.