adding synapse managed vnet feature

deploy/README.md

@@ -48,6 +48,12 @@ Steps 2 through 4 can instead be deployed using a single script below:
 ./deploy/setup.sh <environmentCode> <location> <pipelineName> <envTag>
 
 ```
+To enabled security features like Synapse managed VNET, managed private endpoints, and privates enpoints to 3 synapse endpoints, set 'SECURITY_ENABLED=true' when running setup.sh:
+```
+SECURITY_ENABLED=true ./deploy/setup.sh <environmentCode> <location> <pipelineName> <envTag>
+```
+Before opting in security features, we advise you to get familiar with restrictions we listed below [restrictions-with-security-features](#restrictions-with-security-features)
+
 If you like to package other pipelines or re-package an updated/modified pipeline, follow the instructions under `Packaging the Synapse pipeline` section. The script mentioned in that section can be rerun multiple times.
 
 Arguments | Required | Sample value
@@ -83,6 +89,11 @@ To install infrastructure execute install.sh script as follows
 
 ```
 
+To enabled security features like Synapse managed VNET, managed private endpoints, and privates enpoints to 3 synapse endpoints, set 'SECURITY_ENABLED=true' when running install.sh:
+```
+SECURITY_ENABLED=true ./deploy/install.sh <environmentCode> <location> <envTag>
+```
+
 Default values for the parameters are provided in the script itself.
 
 Arguments | Required | Sample value
@@ -113,7 +124,6 @@ For eg.
 az deployment sub create -l <region> -n aoi -f main.bicep -p location=<region> environmentCode=aoi environment=synapse-aoi
 ```
 
-
 ## Configuring the Resources
 
 If you have deployed the solution using `setup.sh` script, you should skip this step. However, if you have not run the `setup.sh` script, the steps outlined in this section are required.
@@ -142,7 +152,8 @@ Once the above step completes, a zip file is generated. Upload the generated zip
 4. When prompted to select a file, pick the zip file generated in the previous step
 5. Pipelines and its dependencies are imported to the Synapse Studio. Validate the components being imported for any errors
 6. Click "Publish all" and wait for the imported components to be published
-NOTE: You may run into this error during import "Invalid template, please check the template file". It is a known issue that we are working on with the product team. In the interim, we suggest importing from Git Repository as described below.  
+NOTE: You may run into this error during import "Invalid template, please check the template file". It is a known issue that we are working on with the product team. In the interim, we suggest importing from Git Repository as described below.
+
 ## Importing from Git Repository
 
 Another way to get import pipeline into the Synape Studio is through Source Control repository like GitHub or Azure DevOps repository. Refer to the document on [Source Control](https://docs.microsoft.com/azure/synapse-analytics/cicd/source-control) to learn about Git Integration for Azure Synapse Analytics and how to setup.
@@ -304,6 +315,14 @@ To run the pipeline, open the Synapse Studio for the Synapse workspace that you
 
 - Wait for the pipeline to complete.
 
+
+# Restrictions with security features
+- Existing Azure Synapse workspace can not convert to security-enabled one. And, the reverse (i.e. secured one to public one) is not supported either. You cannot change Synapse workspace configuration after the workspace is created. Please refer to [Azure-Synapse-Documentation](https://docs.microsoft.com/en-us/azure/synapse-analytics/security/synapse-workspace-managed-vnet#managed-workspace-virtual-network). 
+- Once SECURITY_ENABLED is turned on during setup/install, Synapse endpoints are restricted to a VNET in the the same region. Thus, you need to create a Windows jumpbox inside the VNET to connect to Synapse Studio. For more details, refer to [synapse-security-diagram](https://docs.microsoft.com/en-us/azure/synapse-analytics/guidance/media/security-white-paper-network-security/private-endpoints.png) and [synapse-network-securiy-section](https://docs.microsoft.com/en-us/azure/synapse-analytics/guidance/security-white-paper-network-security). And, in order to furthur restriction network access to the VNET of your cluster, you also need to restrict the public access of your Synapse workspace. We may achieve this task from azure portal (i.e. Synapse workspace -> Networking -> Public network access -> select 'Disabled').
+- Once security feature is enabled, azure-cli that goes to Azure Synapse like `az synapse managed-private-endpoints` may only work inside the custom VNET and Azure VM attached to it.
+- Once public access to your Synapse workspace is disabled, you may get 'PublicNetworkAccessDenied' error. Enable the public network access for re-deployment and restrict public access afterwards.
+
+
 # Cleanup Script
 
 We have a cleanup script to cleanup the resource groups and thus the resources provisioned using the `environmentCode`.

deploy/addManagedPE.sh

@@ -0,0 +1,147 @@
+#!/usr/bin/env bash
+
+# Copyright (c) Microsoft Corporation.
+# Licensed under the MIT license.
+
+set +x
+if [[ -z "$1" ]]
+  then
+    echo "Environment Code value not supplied"
+    exit 1
+fi
+ENVCODE=$1
+PE_APPROVAL_DESCRIPTION="Approved by addManagedPE.sh script"
+
+approved_managed_private_endpoint_request_exists() {
+    local groupId=$1
+    local resourceName=$2
+    local resourceType=$3
+
+    local peList=$(az network private-endpoint-connection list \
+        -g $groupId -n $resourceName --type $resourceType -ojson 2>/dev/null || echo '')
+    local result=''
+
+    if [[ -n peList ]];
+    then
+        result=$( echo $peList \
+            | jq -r ".[] | select (.properties.privateLinkServiceConnectionState.description == \"${PE_APPROVAL_DESCRIPTION}\").id" )
+        echo $result
+    fi
+    echo $result     
+}
+
+create_synapase_managed_private_endpoint() {
+    local tmpfile=$(mktemp)
+    local synpaseWorkspace=$1
+    local synapseManagedPrivateEndpointName=$2
+    local groupId=$3
+    local privateLinkResourceId=$4
+
+    echo "creating MPE if not exist for $synapseManagedPrivateEndpointName"
+    # check if synapseManagedPrivateEndpointName exists
+    local checkPeExists=$(az synapse managed-private-endpoints show \
+     --pe-name $synapseManagedPrivateEndpointName -otsv --query "id" --workspace-name $synpaseWorkspace 2>/dev/null || echo '')
+
+    if [[ -z $checkPeExists ]];
+    then
+        jq -n -r \
+        --arg groupId "$groupId" \
+        --arg privateLinkResourceId "$privateLinkResourceId" \
+        '{groupId:$groupId, privateLinkResourceId:$privateLinkResourceId}' > $tmpfile
+
+        az synapse managed-private-endpoints create \
+            --file @$tmpfile \
+            --pe-name $synapseManagedPrivateEndpointName \
+            --workspace-name $1
+
+        sleep 60
+        local provisioningState=$(az synapse managed-private-endpoints show --pe-name $synapseManagedPrivateEndpointName \
+            --workspace-name $synpaseWorkspace -o tsv --query "properties.provisioningState")
+        while [[ $provisioningState != "Succeeded" ]];
+        do
+            sleep 10
+            provisioningState=$(az synapse managed-private-endpoints show --pe-name $synapseManagedPrivateEndpointName \
+            --workspace-name $synpaseWorkspace -o tsv --query "properties.provisioningState")
+            echo "provisioningState of $synapseManagedPrivateEndpointName: $provisioningState"
+        done
+    fi    
+}
+
+approve_synapase_managed_private_endpoint() {
+    local resourceGroup=$1
+    local resourceName=$2
+    local resourceType=$3
+
+    local PE_CONNECTION=$(az network private-endpoint-connection list -g $resourceGroup -n $resourceName \
+                    --type $resourceType  --query "[0]" -ojson 2>/dev/null || echo '')
+    if [[ -n $PE_CONNECTION ]];
+    then
+        local PE_CONNECTION_ID=$(echo $PE_CONNECTION | jq -r '.id')
+        local PE_CONNECTION_APPROVAL_STATUS=$(echo $PE_CONNECTION | jq -r '.properties.privateLinkServiceConnectionState.status')
+
+        if [[ $PE_CONNECTION_APPROVAL_STATUS != "Approved" ]];
+        then
+            az network private-endpoint-connection approve \
+                --id $PE_CONNECTION_ID --description "$PE_APPROVAL_DESCRIPTION"
+            echo "$PE_CONNECTION_ID got approved"
+        fi
+    fi
+}
+# wait for SYNAPSE_STORAGE_ACCT showing up in azcli and approve its managed private endpoint first.
+SYNAPSE_STORAGE_ACCT=$(az storage account list --query "[?tags.store && tags.store == 'synapse'].name" -o tsv -g $ENVCODE-pipeline-rg)
+while [[ -z $SYNAPSE_STORAGE_ACCT ]];
+do
+   sleep 30
+   SYNAPSE_STORAGE_ACCT=$(az storage account list --query "[?tags.store && tags.store == 'synapse'].name" -o tsv -g $ENVCODE-pipeline-rg)
+done
+result=$(approved_managed_private_endpoint_request_exists $ENVCODE-pipeline-rg $SYNAPSE_STORAGE_ACCT "Microsoft.Storage/storageAccounts")
+if [[ -z $result ]];
+then
+    approve_synapase_managed_private_endpoint $ENVCODE-pipeline-rg $SYNAPSE_STORAGE_ACCT "Microsoft.Storage/storageAccounts"
+fi
+
+# Create Managed Private Endpoints (PE) if not exist
+PIPELINE_KV=$(az keyvault list --query "[?tags.usage && tags.usage == 'linkedService']" -ojson -g $ENVCODE-pipeline-rg)
+while [[ $PIPELINE_KV == '[]' ]];
+do
+   sleep 30
+   PIPELINE_KV=$(az keyvault list --query "[?tags.usage && tags.usage == 'linkedService']" -ojson -g $ENVCODE-pipeline-rg)
+done
+PIPELINE_KV_NAME=$(echo $PIPELINE_KV | jq -r '.[0].name')
+PIPELINE_KV_ID=$(echo $PIPELINE_KV | jq -r '.[0].id')
+result=$(approved_managed_private_endpoint_request_exists $ENVCODE-pipeline-rg $PIPELINE_KV_NAME "Microsoft.Keyvault/vaults")
+if [[ -z $result ]]
+then
+    create_synapase_managed_private_endpoint "$ENVCODE-pipeline-syn-ws" "$ENVCODE-mpe-pipeline-kv" "vault" "$PIPELINE_KV_ID"
+    approve_synapase_managed_private_endpoint $ENVCODE-pipeline-rg $PIPELINE_KV_NAME "Microsoft.Keyvault/vaults"
+fi 
+
+DATA_STORAGE_ACCT=$(az storage account list --query "[?tags.store && tags.store == 'raw']" -ojson -g $ENVCODE-data-rg)
+while [[ $DATA_STORAGE_ACCT == '[]' ]]
+do
+   sleep 30
+   DATA_STORAGE_ACCT=$(az storage account list --query "[?tags.store && tags.store == 'raw']" -ojson -g $ENVCODE-data-rg)
+done
+DATA_STORAGE_ACCT_NAME=$(echo $DATA_STORAGE_ACCT | jq -r '.[0].name')
+DATA_STORAGE_ACCT_ID=$(echo $DATA_STORAGE_ACCT | jq -r '.[0].id')
+result=$(approved_managed_private_endpoint_request_exists $ENVCODE-data-rg $DATA_STORAGE_ACCT_NAME "Microsoft.Storage/storageAccounts")
+if [[ -z $result ]]
+then
+    create_synapase_managed_private_endpoint "$ENVCODE-pipeline-syn-ws" "$ENVCODE-mpe-data-raw" "dfs" "$DATA_STORAGE_ACCT_ID"
+    approve_synapase_managed_private_endpoint $ENVCODE-data-rg $DATA_STORAGE_ACCT_NAME "Microsoft.Storage/storageAccounts"
+fi
+
+DATA_KV=$(az keyvault list --query "[?tags.usage && tags.usage == 'general']" -ojson -g $ENVCODE-data-rg)
+while [[ $DATA_KV == '[]' ]];
+do
+    sleep 30
+    DATA_KV=$(az keyvault list --query "[?tags.usage && tags.usage == 'general']" -ojson -g $ENVCODE-data-rg)
+done
+DATA_KV_NAME=$(echo $DATA_KV | jq -r '.[0].name')
+DATA_KV_ID=$(echo $DATA_KV | jq -r '.[0].id')
+result=$(approved_managed_private_endpoint_request_exists $ENVCODE-data-rg $DATA_KV_NAME "Microsoft.Keyvault/vaults")
+if [[ -z $result ]]
+then
+    create_synapase_managed_private_endpoint "$ENVCODE-pipeline-syn-ws" "$ENVCODE-mpe-data-kv" "vault" "$DATA_KV_ID"
+    approve_synapase_managed_private_endpoint $ENVCODE-data-rg $DATA_KV_NAME "Microsoft.Keyvault/vaults"
+fi

deploy/infra/groups/pipeline.bicep

@@ -84,6 +84,8 @@ param synapseMIStorageAccountRoles array = [
 ]
 
 param logAnalyticsWorkspaceId string
+param securityEnabled bool = false
+param preventDataExfiltration bool = false
 
 var namingPrefix = '${environmentCode}-${projectName}'
 var synapseResourceGroupNameVar = empty(synapseResourceGroupName) ? '${namingPrefix}-rg' : synapseResourceGroupName
@@ -129,8 +131,6 @@ module synapseHnsStorageAccount '../modules/storage.hns.bicep' = {
   }
 }
 
-
-
 module synapseWorkspace '../modules/synapse.workspace.bicep' = {
   name: '${namingPrefix}-workspace'
   params:{
@@ -153,6 +153,8 @@ module synapseWorkspace '../modules/synapse.workspace.bicep' = {
     gitRepoRootFolder: synapseGitRepoRootFolder
     gitRepoVstsTenantId: synapseGitRepoVstsTenantId
     gitRepoType: synapseGitRepoType
+    createManagedVnet: securityEnabled
+    preventDataExfiltration: preventDataExfiltration
   }
   dependsOn: [
     synapseHnsStorageAccount

deploy/infra/main.bicep

@@ -14,6 +14,15 @@ param environmentCode string
 @description('Environment will be used as Tag on the resource group')
 param environment string
 
+@description('Flag to set whether security resources such as Synapse managed vnet, NSG, etc are created or not')
+param securityEnabled bool = false
+
+@description('preventDataExfiltration for Synapse managed vnet')
+param preventDataExfiltration bool = false
+
+@description('Flag to determine whether VNET and Subnet will be created or re-created')
+param createVnetSubnets bool = true
+
 @description('Used for naming of the network resource group and its resources')
 param networkModulePrefix string = 'network'
 
@@ -45,7 +54,7 @@ module networkResourceGroup 'modules/resourcegroup.bicep' = {
   }
 }
 
-module networkModule 'groups/networking.bicep' = {
+module networkModule 'groups/networking.bicep' = if (createVnetSubnets) {
   name: '${networkModulePrefix}-module'
   scope: resourceGroup(networkResourceGroup.name)
   params: {
@@ -105,6 +114,8 @@ module pipelineModule 'groups/pipeline.bicep' = {
     environmentCode: environmentCode
     environmentTag: environment
     logAnalyticsWorkspaceId: monitorModule.outputs.workspaceId
+    securityEnabled: securityEnabled
+    preventDataExfiltration: preventDataExfiltration
   }
   dependsOn: [
     networkModule

deploy/infra/modules/privateendpoints.bicep

@@ -0,0 +1,60 @@
+// Copyright (c) Microsoft Corporation.
+// Licensed under the MIT license.
+
+param environmentCode string
+param location string
+param subnetId string
+param privateLinkServiceId string
+param privateDnsZoneName string
+param groupIds array
+
+resource privateDnsZone 'Microsoft.Network/privateDnsZones@2020-06-01' existing = {
+  name: privateDnsZoneName
+}
+
+resource privateDnsZoneLink 'Microsoft.Network/privateDnsZones/virtualNetworkLinks@2020-06-01' existing = {
+  parent: privateDnsZone
+  name: privateDnsZoneName
+}
+
+resource privateEndpoints 'Microsoft.Network/privateEndpoints@2021-05-01' = {
+  name: guid(environmentCode, privateLinkServiceId, groupIds[0])
+  location: location
+  dependsOn: [
+    privateDnsZoneLink
+  ]
+  properties: {
+    subnet: {
+      id: subnetId
+    }
+    privateLinkServiceConnections: [
+      {
+        name: guid(environmentCode, privateLinkServiceId, groupIds[0])
+        properties: {
+          privateLinkServiceId: privateLinkServiceId
+          groupIds: groupIds
+          privateLinkServiceConnectionState: {
+            status: 'Approved'
+            description: 'Auto-Approved'
+            actionsRequired: 'None'
+          }
+        }
+      }
+    ]
+  }
+}
+
+resource privateDnsZoneGroup 'Microsoft.Network/privateEndpoints/privateDnsZoneGroups@2021-05-01' = {
+  parent: privateEndpoints
+  name: guid(environmentCode, privateLinkServiceId, groupIds[0])
+  properties: {
+    privateDnsZoneConfigs: [
+      {
+        name: privateDnsZone.name
+        properties: {
+          privateDnsZoneId: privateDnsZone.id
+        }
+      }
+    ]
+  }
+}

deploy/infra/modules/privatelink.bicep

@@ -0,0 +1,23 @@
+// Copyright (c) Microsoft Corporation.
+// Licensed under the MIT license.
+
+param privateDnsZoneName string
+param customVnetId string
+
+resource privateDnsZone 'Microsoft.Network/privateDnsZones@2020-06-01' = {
+  name: privateDnsZoneName
+  location: 'global'
+}
+
+resource privateDnsZoneLink 'Microsoft.Network/privateDnsZones/virtualNetworkLinks@2020-06-01' = {
+  parent: privateDnsZone
+  name: privateDnsZoneName
+  location: 'global'
+  properties : {
+    registrationEnabled: false
+    virtualNetwork: {
+      id: customVnetId
+    }
+  }
+}
+