feat: added support to create KMS auth policy (#313)

Author: jor2

README.md

@@ -12,10 +12,6 @@ This module implements IBM Event Streams for IBM Cloud with topics, partitions,
 
 The Event Streams service supports payload data encryption that uses a root key CRN of a key management service, such as Key Protect or Hyper Protect Crypto Services. You specify the root key CRN with the `kms_key_crn` input. For more information, see [Managing encryption in Event Streams](https://cloud.ibm.com/docs/EventStreams?topic=EventStreams-managing_encryption).
 
-Before you run the module, configure an authorization policy to allow the Event Streams service to access the key management service instance with the reader role. For more information, see [Using authorizations to grant access between services](https://cloud.ibm.com/docs/account?topic=account-serviceauth).
-
-You can't manage the policy in the same Terraform state file as the Event Streams service instance. When you issue a `terraform destroy` command, the instance is only soft deleted and remains as a reclamation resource for a while to support recovery (reclamation). An authorization policy must exist when the instance is hard deleted or reclaimed or else the unregistration of the instance from the root key fails on the backend. If the policy doesn't exist, the only way to unregister the instance, which is a requirement for deletion of the root key, is by opening a support case. For more information, see [Using a customer-managed key](https://cloud.ibm.com/docs/EventStreams?topic=EventStreams-managing_encryption#using_encryption).
-
 <!-- Below content is automatically populated via pre-commit hook -->
 <!-- BEGIN OVERVIEW HOOK -->
 ## Overview
@@ -25,6 +21,7 @@ You can't manage the policy in the same Terraform state file as the Event Stream
 * [Examples](./examples)
     * [Basic example](./examples/basic)
     * [Complete example with topics and schema creation.](./examples/complete)
+    * [Financial Services Cloud profile example](./examples/fscloud)
 * [Contributing](#contributing)
 <!-- END OVERVIEW HOOK -->
 
@@ -110,6 +107,7 @@ You need the following permissions to run this module.
 |------|---------|
 | <a name="requirement_terraform"></a> [terraform](#requirement\_terraform) | >= 1.3.0 |
 | <a name="requirement_ibm"></a> [ibm](#requirement\_ibm) | >= 1.65.0, <2.0.0 |
+| <a name="requirement_time"></a> [time](#requirement\_time) | >= 0.9.1 |
 
 ### Modules
 
@@ -123,7 +121,9 @@ You need the following permissions to run this module.
 |------|------|
 | [ibm_event_streams_schema.es_schema](https://registry.terraform.io/providers/IBM-Cloud/ibm/latest/docs/resources/event_streams_schema) | resource |
 | [ibm_event_streams_topic.es_topic](https://registry.terraform.io/providers/IBM-Cloud/ibm/latest/docs/resources/event_streams_topic) | resource |
+| [ibm_iam_authorization_policy.kms_policy](https://registry.terraform.io/providers/IBM-Cloud/ibm/latest/docs/resources/iam_authorization_policy) | resource |
 | [ibm_resource_instance.es_instance](https://registry.terraform.io/providers/IBM-Cloud/ibm/latest/docs/resources/resource_instance) | resource |
+| [time_sleep.wait_for_authorization_policy](https://registry.terraform.io/providers/hashicorp/time/latest/docs/resources/sleep) | resource |
 
 ### Inputs
 
@@ -133,12 +133,15 @@ You need the following permissions to run this module.
 | <a name="input_create_timeout"></a> [create\_timeout](#input\_create\_timeout) | The timeout value for creating an Event Streams instance. Specify `3h` for an Enterprise plan instance. Add 1 h for each level of non-default throughput. Add 30 min for each level of non-default storage size. | `string` | `"3h"` | no |
 | <a name="input_delete_timeout"></a> [delete\_timeout](#input\_delete\_timeout) | The timeout value for deleting an Event Streams instance. | `string` | `"15m"` | no |
 | <a name="input_es_name"></a> [es\_name](#input\_es\_name) | The name to give the Event Streams instance created by this module. | `string` | n/a | yes |
-| <a name="input_kms_key_crn"></a> [kms\_key\_crn](#input\_kms\_key\_crn) | The root key CRN of the key management service (Key Protect or Hyper Protect Crypto Services) to use to encrypt the payload data. [Learn more](https://cloud.ibm.com/docs/EventStreams?topic=EventStreams-managing_encryption) about integrating Hyper Protect Crypto Services with Event Streams. Configure an authorization policy to allow the Event Streams service to access the key management service instance with the reader role ([Learn more](https://cloud.ibm.com/docs/account?topic=account-serviceauth)). You can't manage the policy in the same Terraform state file as the Event Streams service instance ([Learn more](https://cloud.ibm.com/docs/EventStreams?topic=EventStreams-managing_encryption#using\_encryption)). | `string` | `null` | no |
+| <a name="input_existing_kms_instance_guid"></a> [existing\_kms\_instance\_guid](#input\_existing\_kms\_instance\_guid) | The GUID of the Hyper Protect Crypto Services or Key Protect instance in which the key specified in var.kms\_key\_crn is coming from. Required only if var.kms\_encryption\_enabled is set to true, var.skip\_iam\_authorization\_policy is set to false, and you pass a value for var.kms\_key\_crn. | `string` | `null` | no |
+| <a name="input_kms_encryption_enabled"></a> [kms\_encryption\_enabled](#input\_kms\_encryption\_enabled) | Set this to true to control the encryption keys used to encrypt the data that you store in IBM Cloud® Databases. If set to false, the data is encrypted by using randomly generated keys. For more info on Key Protect integration, see https://cloud.ibm.com/docs/cloud-databases?topic=cloud-databases-key-protect. For more info on HPCS integration, see https://cloud.ibm.com/docs/cloud-databases?topic=cloud-databases-hpcs | `bool` | `false` | no |
+| <a name="input_kms_key_crn"></a> [kms\_key\_crn](#input\_kms\_key\_crn) | The root key CRN of the key management service (Key Protect or Hyper Protect Crypto Services) to use to encrypt the payload data. [Learn more](https://cloud.ibm.com/docs/EventStreams?topic=EventStreams-managing_encryption) about integrating Hyper Protect Crypto Services with Event Streams. | `string` | `null` | no |
 | <a name="input_plan"></a> [plan](#input\_plan) | The plan for the Event Streams instance. Possible values: `lite`, `standard`, `enterprise-3nodes-2tb`. | `string` | `"standard"` | no |
 | <a name="input_region"></a> [region](#input\_region) | The region where the Event Streams are created. | `string` | `"us-south"` | no |
 | <a name="input_resource_group_id"></a> [resource\_group\_id](#input\_resource\_group\_id) | The resource group ID where the Event Streams instance is created. | `string` | n/a | yes |
 | <a name="input_schemas"></a> [schemas](#input\_schemas) | The list of schema objects. Include the `schema_id` and the `type` and `name` of the schema in the `schema` object. | <pre>list(object(<br/>    {<br/>      schema_id = string<br/>      schema = object({<br/>        type = string<br/>        name = string<br/>        fields = optional(list(object({<br/>          name = string<br/>          type = string<br/>        })))<br/>      })<br/>    }<br/>  ))</pre> | `[]` | no |
 | <a name="input_service_endpoints"></a> [service\_endpoints](#input\_service\_endpoints) | The type of service endpoints. Possible values: 'public', 'private', 'public-and-private'. | `string` | `"public"` | no |
+| <a name="input_skip_iam_authorization_policy"></a> [skip\_iam\_authorization\_policy](#input\_skip\_iam\_authorization\_policy) | Set to true to skip the creation of an IAM authorization policy that permits all Event Streams database instances in the resource group to read the encryption key from the KMS instance. If set to false, pass in a value for the KMS instance in the existing\_kms\_instance\_guid variable. In addition, no policy is created if var.kms\_encryption\_enabled is set to false. | `bool` | `false` | no |
 | <a name="input_storage_size"></a> [storage\_size](#input\_storage\_size) | Storage size of the Event Streams in GB. Applies only to Enterprise plan instances. Possible values: `2048`, `4096`, `6144`, `8192`, `10240`, `12288`. Storage capacity cannot be reduced after the instance is created. When the `throughput` input variable is set to `300`, storage size starts at 4096. When `throughput` is `450`, storage size starts starts at `6144`. | `number` | `"2048"` | no |
 | <a name="input_tags"></a> [tags](#input\_tags) | The list of tags associated with the Event Steams instance. | `list(string)` | `[]` | no |
 | <a name="input_throughput"></a> [throughput](#input\_throughput) | Throughput capacity in MB per second. Applies only to Enterprise plan instances. Possible values: `150`, `300`, `450`. | `number` | `"150"` | no |

examples/fscloud/README.md

@@ -0,0 +1,17 @@
+ # Financial Services Cloud profile example
+
+An end-to-end example that uses the [Profile for IBM Cloud Framework for Financial Services](https://github.com/terraform-ibm-modules/terraform-ibm-event-streams/tree/main/modules/fscloud) to deploy an instance of Event Streams.
+
+The example uses the IBM Cloud Terraform provider to create the following infrastructure:
+
+- A resource group, if one is not passed in.
+- An IAM authorization between all Event Stream instances in the given resource group and the Hyper Protect Crypto Services instance that is passed in.
+- An Event Streams instance that is encrypted with the Hyper Protect Crypto Services root key that is passed in.
+- A sample virtual private cloud (VPC).
+- A context-based restriction (CBR) rule to only allow Event Streams to be accessible from within the VPC.
+
+:exclamation: **Important:** In this example, only the Event Streams instance complies with the IBM Cloud Framework for Financial Services. Other parts of the infrastructure do not necessarily comply.
+
+## Before you begin
+
+- You need a Hyper Protect Crypto Services instance and root key available in the region that you want to deploy your Event Streams instance to.

examples/fscloud/main.tf

@@ -0,0 +1,83 @@
+##############################################################################
+# Resource Group
+##############################################################################
+
+module "resource_group" {
+  source = "git::https://github.com/terraform-ibm-modules/terraform-ibm-resource-group.git?ref=v1.1.6"
+  # if an existing resource group is not set (null) create a new one using prefix
+  resource_group_name          = var.resource_group == null ? "${var.prefix}-resource-group" : null
+  existing_resource_group_name = var.resource_group
+}
+
+##############################################################################
+# Get Cloud Account ID
+##############################################################################
+
+data "ibm_iam_account_settings" "iam_account_settings" {
+}
+
+##############################################################################
+# VPC
+##############################################################################
+resource "ibm_is_vpc" "example_vpc" {
+  name           = "${var.prefix}-vpc"
+  resource_group = module.resource_group.resource_group_id
+  tags           = var.resource_tags
+}
+
+resource "ibm_is_subnet" "testacc_subnet" {
+  name                     = "${var.prefix}-subnet"
+  vpc                      = ibm_is_vpc.example_vpc.id
+  zone                     = "${var.region}-1"
+  total_ipv4_address_count = 256
+  resource_group           = module.resource_group.resource_group_id
+}
+
+##############################################################################
+# Create CBR Zone
+##############################################################################
+module "cbr_zone" {
+  source           = "terraform-ibm-modules/cbr/ibm//modules/cbr-zone-module"
+  version          = "1.27.0"
+  name             = "${var.prefix}-VPC-network-zone"
+  zone_description = "CBR Network zone representing VPC"
+  account_id       = data.ibm_iam_account_settings.iam_account_settings.account_id
+  addresses = [{
+    type  = "vpc", # to bind a specific vpc to the zone
+    value = ibm_is_vpc.example_vpc.crn,
+  }]
+}
+
+
+# #############################################################################
+# Events-streams-instance
+# #############################################################################
+
+module "event_streams" {
+  source                     = "../../modules/fscloud"
+  resource_group_id          = module.resource_group.resource_group_id
+  es_name                    = "${var.prefix}-es-fs"
+  kms_key_crn                = var.kms_key_crn
+  schemas                    = var.schemas
+  tags                       = var.resource_tags
+  topics                     = var.topics
+  existing_kms_instance_guid = var.existing_kms_instance_guid
+  cbr_rules = [
+    {
+      description      = "${var.prefix}-event stream access only from vpc"
+      enforcement_mode = "enabled"
+      account_id       = data.ibm_iam_account_settings.iam_account_settings.account_id
+      rule_contexts = [{
+        attributes = [
+          {
+            "name" : "endpointType",
+            "value" : "private"
+          },
+          {
+            name  = "networkZoneId"
+            value = module.cbr_zone.zone_id
+        }]
+      }]
+    }
+  ]
+}

examples/fscloud/outputs.tf

@@ -0,0 +1,33 @@
+##############################################################################
+# Outputs
+##############################################################################
+
+output "resource_group_name" {
+  description = "Resource group name"
+  value       = module.resource_group.resource_group_name
+}
+
+output "resource_group_id" {
+  description = "Resource group ID"
+  value       = module.resource_group.resource_group_id
+}
+
+output "crn" {
+  description = "Event Streams instance crn"
+  value       = module.event_streams.crn
+}
+
+output "guid" {
+  description = "Event Streams instance guid"
+  value       = module.event_streams.guid
+}
+
+output "kafka_brokers_sasl" {
+  description = "(Array of Strings) Kafka brokers use for interacting with Kafka native API"
+  value       = module.event_streams.kafka_brokers_sasl
+}
+
+output "kafka_http_url" {
+  description = "The API endpoint to interact with Event Streams REST API"
+  value       = module.event_streams.kafka_http_url
+}

examples/fscloud/provider.tf

@@ -0,0 +1,4 @@
+provider "ibm" {
+  ibmcloud_api_key = var.ibmcloud_api_key
+  region           = var.region
+}

examples/fscloud/variables.tf

@@ -0,0 +1,65 @@
+variable "ibmcloud_api_key" {
+  type        = string
+  description = "The IBM Cloud API Key"
+  sensitive   = true
+}
+
+variable "region" {
+  type        = string
+  description = "Region to provision all resources created by this example"
+  default     = "us-south"
+}
+
+variable "prefix" {
+  type        = string
+  description = "Prefix to append to all resources created by this example"
+  default     = "fs-cloud"
+}
+
+variable "resource_group" {
+  type        = string
+  description = "An existing resource group name to use for this example, if unset a new resource group will be created"
+  default     = null
+}
+
+variable "resource_tags" {
+  type        = list(string)
+  description = "List of tags associated with the Event Steams instance"
+  default     = []
+}
+
+variable "schemas" {
+  type = list(object(
+    {
+      schema_id = string
+      schema = object({
+        type = string
+        name = string
+      })
+    }
+  ))
+  description = "The list of schema object which contains schema id and format of the schema"
+  default     = []
+}
+
+variable "topics" {
+  type = list(object(
+    {
+      name       = string
+      partitions = number
+      config     = object({})
+    }
+  ))
+  description = "List of topics. For lite plan only one topic is allowed."
+  default     = []
+}
+
+variable "existing_kms_instance_guid" {
+  description = "The GUID of the Hyper Protect Crypto service in which the key specified in var.kms_key_crn is coming from"
+  type        = string
+}
+
+variable "kms_key_crn" {
+  type        = string
+  description = "The root key CRN of a Hyper Protect Crypto Service (HPCS) that you want to use for disk encryption. See https://cloud.ibm.com/docs/cloud-databases?topic=cloud-databases-hpcs&interface=ui for more information on integrating HPCS with Event Streams instance."
+}

examples/fscloud/version.tf

@@ -0,0 +1,10 @@
+terraform {
+  required_version = ">= 1.3.0"
+  required_providers {
+    # Use latest version of provider in non-basic examples to verify latest version works with module
+    ibm = {
+      source  = "IBM-Cloud/ibm"
+      version = ">= 1.56.1"
+    }
+  }
+}

main.tf

@@ -7,15 +7,34 @@ locals {
   # tflint-ignore: terraform_unused_declarations
   validate_kms_plan = var.kms_key_crn != null && var.plan != "enterprise-3nodes-2tb" ? tobool("kms encryption is only supported for enterprise plan") : true
   # tflint-ignore: terraform_unused_declarations
+  validate_kms_values = !var.kms_encryption_enabled && var.kms_key_crn != null ? tobool("When passing values for var.kms_key_crn, you must set var.kms_encryption_enabled to true. Otherwise unset them to use default encryption") : true
+  # tflint-ignore: terraform_unused_declarations
+  validate_kms_vars = var.kms_encryption_enabled && var.kms_key_crn == null ? tobool("When setting var.kms_encryption_enabled to true, a value must be passed for var.kms_key_crn and/or var.backup_encryption_key_crn") : true
+  # tflint-ignore: terraform_unused_declarations
+  validate_auth_policy = var.kms_encryption_enabled && var.skip_iam_authorization_policy == false && var.existing_kms_instance_guid == null ? tobool("When var.skip_iam_authorization_policy is set to false, and var.kms_encryption_enabled to true, a value must be passed for var.existing_kms_instance_guid in order to create the auth policy.") : true
+  # tflint-ignore: terraform_unused_declarations
   validate_throughput_lite_standard = ((var.plan == "lite" || var.plan == "standard") && var.throughput != 150) ? tobool("Throughput value cannot be changed in lite and standard plan. Default value is 150.") : true
   # tflint-ignore: terraform_unused_declarations
   validate_storage_size_lite_standard = ((var.plan == "lite" || var.plan == "standard") && var.storage_size != 2048) ? tobool("Storage size value cannot be changed in lite and standard plan. Default value is 2048.") : true
   # tflint-ignore: terraform_unused_declarations
   validate_service_end_points_lite_standard = ((var.plan == "lite" || var.plan == "standard") && var.service_endpoints != "public") ? tobool("Service endpoint cannot be changed in lite and standard plan. Default is public.") : true
+  # Determine what KMS service is being used for database encryption
+  kms_service = var.kms_key_crn != null ? (
+    can(regex(".*kms.*", var.kms_key_crn)) ? "kms" : (
+      can(regex(".*hs-crypto.*", var.kms_key_crn)) ? "hs-crypto" : null
+    )
+  ) : null
+}
+
+# workaround for https://github.com/IBM-Cloud/terraform-provider-ibm/issues/4478
+resource "time_sleep" "wait_for_authorization_policy" {
+  depends_on = [ibm_iam_authorization_policy.kms_policy]
 
+  create_duration = "30s"
 }
 
 resource "ibm_resource_instance" "es_instance" {
+  depends_on        = [time_sleep.wait_for_authorization_policy]
   name              = var.es_name
   service           = "messagehub"
   plan              = var.plan
@@ -59,6 +78,20 @@ resource "ibm_event_streams_topic" "es_topic" {
   config               = var.topics[count.index].config
 }
 
+##############################################################################
+# IAM Authorization Policy
+##############################################################################
+
+# Create IAM Authorization Policies to allow messagehub to access kms for the encryption key
+resource "ibm_iam_authorization_policy" "kms_policy" {
+  count                       = var.kms_encryption_enabled == false || var.skip_iam_authorization_policy ? 0 : 1
+  source_service_name         = "messagehub"
+  source_resource_group_id    = var.resource_group_id
+  target_service_name         = local.kms_service
+  target_resource_instance_id = var.existing_kms_instance_guid
+  roles                       = ["Reader"]
+  description                 = "Allow all Event Streams instances in the resource group ${var.resource_group_id} to read from the ${local.kms_service} instance GUID ${var.existing_kms_instance_guid}"
+}
 
 ##############################################################################
 # Context Based Restrictions