Integration of Web Transactions from Netskope Log Streaming to Microsoft Sentinel — Azure Government Cloud

 This document outlines the integration of Netskope Web Transaction Logs with Microsoft Sentinel on Azure US Government cloud using the Codeless Connector Framework (CCF). Separate templates are provided for comma-delimited and space-delimited blob data. This native integration provides near real-time visibility for effective threat hunting and incident management. To configure Netskope Log Streaming to send logs to your Blob storage please go through the article: https://docs.netskope.com/en/stream-logs-to-azure-blob

⚠️ Important: You can't restrict network access to your Azure Blob Storage account as per the official guidance from Microsoft.

🏗️ Integration Architecture Overview

The integration leverages several Azure services to ensure reliable and scalable log delivery:

1. Netskope to Azure Storage Blob: Netskope pushes Web Transaction Logs into a customer-owned Azure Storage Blob Container (as gzipped CSV files).

2. Event Grid Notification: Every time a new file (blob) is written, an automatic notification is sent to Azure Event Grid.

3. Storage Queue Messaging: Event Grid pushes the blob URI to a Storage Queue. This acts as a work queue for the connector.

4. Microsoft Sentinel CCF Connector (Scuba Workers): These connectors continuously pull (pop) messages from the Storage Queue.

5. Data Ingestion: The Scuba workers retrieve the data from the Blob Storage, process it, and ingest it into the Log Analytics Workspace via a Data Collection Rule (DCR).

6. Log Analytics: The Netskope Web Transactions appear in the NetskopeWebTransactions_CL table within Microsoft Sentinel.

🔀 Comma vs Space Delimited Templates

Netskope can write Web Transaction blobs as either comma-delimited (standard CSV) or space-delimited (W3C Extended Log File Format). Because the Sentinel CCF blob connector parses CSV using a single delimiter character, two separate Government-cloud templates are provided. Both write to the same NetskopeWebTransactions_CL table, so downstream queries, workbooks, and analytics rules are identical.

Deploy only the one that matches your Netskope output. To confirm which delimiter your tenant emits, download a single blob and inspect the first line — comma-separated fields → use the Comma template; space-separated fields → use the Space template.

🔒 Prerequisites

To successfully deploy the custom template and configure the Netskope Data Connector, the user or Service Principal performing the steps must have the following minimum permissions assigned:

1. Configure Netskope Log Streaming to Azure

Follow the documentation: Stream Logs to Azure Blob - Netskope Knowledge Portal for the Netskope Log Streaming configuration to Azure.

2. Microsoft Entra ID (Tenant-Level Role)

The deployment process involves working with Enterprise Applications (Service Principals) to set up the connector.

3. Azure Role-Based Access Control (Subscription-Level Role)

The connector deployment uses an ARM template that creates several Azure resources (Data Collection Rule, Event Subscription, etc.) and requires the ability to deploy resources.

4. Microsoft Sentinel-Enabled Log Analytics Workspace

You must have a Log Analytics workspace with Microsoft Sentinel enabled. If you don't have one, create it first: Onboard to Microsoft Sentinel | Microsoft Learn.

5. Enable Auditing & Health Monitoring

Make sure to turn on Auditing & Health Monitoring for the Azure Sentinel: https://learn.microsoft.com/en-us/azure/sentinel/enable-monitoring. This will help us to get to the root cause of the issue.

6. Confirm the Workspace Region (Government cloud)

The Data Collection Rule must be created in the same region as the Log Analytics workspace. Deploying with a mismatched Workspace-location results in an InvalidWorkspace error at connect time. Confirm your workspace region before deploying and use that value for the Workspace-location parameter.

How to Enable the Integration

1. Create the Service Principal

The connector authenticates using a Microsoft first-party application. Its Service Principal must exist in your tenant before connecting. Create it with the Azure CLI (signed in to your Government tenant, with az cloud set --name AzureUSGovernment):

az ad sp create --id e648f588-ed96-40ee-8ffa-12e13d6b16fa

If it returns "already in use", the Service Principal already exists — no action needed. Either way, the Enterprise Application (display name ScubaSentinelToStorageUSGov) will be present in your tenant, and its Object ID auto-populates the connector's Service Principal ID field.

Provide the Admin consent to the particular Application
 

2. Deploy the Azure Sentinel Connector Template

1. Navigate to the Azure Government Portal.
2. Search for and select Deploy a Custom Template.
3. Select Build your own template in the editor.
4. Paste the contents of the template that matches your data (from the GitHub links in the table above) into the editor.
5. Click Next.
6. Fill in the required deployment parameters (Workspace, Workspace-location, Subscription, and Resource Group where your Sentinel instance is located).
7. Click Review + Create and then Create to deploy the necessary Azure resources (Event Subscription, Storage Queues, Data Collection Rule, etc.).

3. Configure the Data Connector in Sentinel

1. Navigate to your Microsoft Sentinel Instance.
2. Under Content management, select Data Connectors.
3. Locate the new connector named NetskopeWebTxConnector.
4. Select Open connector page.
5. Fill out the required parameters:

6. Before clicking on connect assign the Storage Blob Data Contributor and Storage Queue Data Contributor at the Service Principal Level.

7. Click Connect to finalize the integration.

📝 Note: Data will typically begin appearing in your Log Analytics workspace within 20 minutes after successful connection.

8. After connection you can check out the data in the Table: NetskopeWebTransactions_CL

Workbook & Analytics Rules

For Workbook and Analytics Rules creation, please follow the corresponding article.

Troubleshooting

1. Perform all the steps below first in a "clean room" environment. That is, create a new resource group which contains only the minimal number of resources: Sentinel-enabled Log Analytics workspace, a Storage Account, and a Blob Container with expected name.
2. Ensure diagnostics logs are enabled on the Sentinel Workspace. Do this before anything else.
3. Deploy ONLY ONE Netskope Log Streaming Blob solution template into this workspace. Check resource group deployments to ensure that it deployed without error. After deploying, if you make any changes to the solution template, start over from step 1 in a new clean resource group.
4. Connect the connector. Use the storage account in the same resource group as the Sentinel Workspace. In case of the Netskope connector, leave event grid topic blank, and be sure to set a folder name.
5. Check resource group deployments to ensure that it deployed without error.
6. Check that the storage account has a "Microsoft.Storage.BlobCreated" Event Subscription, with expected prefix folder.
7. Click into the Event Subscription details. Ensure the "Endpoint" is {name}-notification.
8. Check that the Storage account has two queues, {name}-dlq and {name}-notification.
9. Check the role assignments on each queue. Ensure the expected App Registration (e.g. "ScubaSentinelToStorageUSGov") has the Storage Queue Data Contributor role. You can verify it's the correct app by clicking on its name, and comparing the Object ID on the "Enterprise Application Overview" page with the Service Principal ID in the connector deployment page.
10. Manually add a piece of test data to the blob container. Be sure the data is formatted correctly, as the DCR will expect it, is in the correct format (e.g. .csv.gz) with the correct delimiter, and is in the expected folder.
11. Ensure the blob was added in the correct folder, with the expected name.
12. Check the metrics on the Events page for the Storage Account. Look for Published Events and Delivered Events to jump up to 1. Refresh as needed.
13. Check the {name}-notification queue quickly. Look for a message to arrive in that queue with the folder and file name matching the file you just added. Refresh regularly for a few minutes, could be up to 10. The event should disappear from the queue because it has been picked up by the connector.
14. Check the metrics for the DCR. Look for "Log Ingestion Requests per minute" to come up.
15. Check the Sentinel Logs, run a simple KQL query for just the table name. Look for matching data to come up.
16. Visit the connector page. Look for green status, and indication that "last log received" was recent.
17. Repeat steps 10-15 a few more times.
18. Now turn on your blob data source, and repeat steps 11-15.
19. Problems? Check the SentinelHealth table for clues.

Common Errors

• Deployment Failed — { "error": { "code": "BadRequest", "message": "Connectivity check failed. ConnectorId: NetskopeWebTransactions, Status code:S3B40010, Message:An unknown exception occurred." } } — The issue is mainly due to a missing role assignment to the Service Principal, the Service Principal not being created properly, or Blob storage being enabled from a selected network where Scuba IPs are not whitelisted.

• Failed to create required resources for data connector — Invalid output table schema {0}: The following columns which exist in the current schema do not exist in the new schema or have different types : {1} — This kind of error is seen when you have made changes to the template schema or directly in your environment.

• { "code": "BadRequest", "message": "System topic source cannot be modified." } — Your system topic is already existing. Copy the existing system topic name from the Blob Storage > Events.

• InvalidWorkspace: Log Analytics destination resource ... is in different location ... than the Data Collection Rule — The Workspace-location you deployed with doesn't match the workspace's actual region. Confirm the region and redeploy with the matching value.

• Rows ingest but every column except TimeGenerated is empty — The wrong delimiter template was deployed. Confirm the blob's actual delimiter and deploy the matching template.

Change Logs:

• Added Government-cloud (*.usgovcloudapi.net) templates with separate comma-delimited and space-delimited variants, both writing to NetskopeWebTransactions_CL.
• Documented the Service Principal creation command for the connector app.
• Added a workspace-region confirmation step and the InvalidWorkspace error to Common Errors, as region mismatch is the most common Government-cloud failure.