Terraform Support for Data Collection

data-collection/deploy/terraform/README.md

@@ -0,0 +1,114 @@
+# Cloud Intelligence Dashboards - Terraform Deployment Solution
+
+## Overview
+
+This Terraform solution simplifies the deployment of data collection components for Cloud Intelligence Dashboards across AWS Organizations. It provides a streamlined approach to configure and manage data collection from multiple AWS accounts without relying on StackSets, offering better control and flexibility over the deployment process.
+
+The solution acts as a Terraform wrapper around AWS CloudFormation templates, providing a more manageable and maintainable infrastructure-as-code approach for enterprise environments.
+
+## Architecture
+
+The solution consists of three main Terraform modules that work together to enable comprehensive data collection:
+
+### 1. Management Account Module
+
+This module configures the necessary resources in the AWS Organizations management account:
+
+- Creates IAM roles with specific permissions for monitoring and data collection
+- Enables required AWS services and APIs
+- Configures access for specific features like AWS Backup, Compute Optimizer, and Cost Anomaly Detection
+
+### 2. Linked Account Role Module
+
+This module deploys the necessary IAM roles to each linked account for monitoring purposes:
+
+- Creates standardized IAM roles across member accounts
+- Configures granular permissions for various data collection features
+
+### 3. Data Collection Module
+
+This module orchestrates the data collection process and deploys CloudFormation template for data collection as a wrapper
+
+## Version Locking
+
+For production deployments, you should lock module versions to a release tag to better control when and what updates are made.
+Use "*tag_version*" variable to select a release tag.
+```bash
+tag_version = "3.6.2"
+```
+
+For a complete list of release tags, visit [https://github.com/awslabs/cid-data-collection-framework/tags]https://github.com/awslabs/cid-data-collection-framework/tags.
+
+## Deployment Process
+
+### Step 1: Deploy Management Account Module
+
+1. Configure your AWS credentials for the management account
+2. Initialize and apply the management account module:
+```bash
+cd management-accounts
+terraform init
+terraform plan
+terraform apply
+```
+
+### Step 2: Deploy Linked Account Roles
+
+1. For each linked account:
+   - Configure AWS credentials
+   - Initialize and apply the linked account role module
+```bash
+cd linked-accounts
+terraform init
+terraform plan
+terraform apply
+```
+
+### Step 3: Deploy Data Collection
+
+1. Use the outputs from previous modules as inputs
+2. Deploy the data collection module:
+```bash
+cd data-collection
+terraform init
+terraform plan
+terraform apply
+```
+
+## Why Not StackSets?
+
+This solution intentionally avoids using AWS StackSets for several reasons:
+- Greater control over deployment timing and sequencing
+- Simplified troubleshooting and rollback procedures
+- Better integration with existing Terraform workflows
+- More flexible permission management
+- Enhanced visibility of deployment status per account
+
+## Troubleshooting
+
+Common issues and solutions:
+
+1. **Linked Account Access Issues**
+   - Verify IAM role trust relationships
+   - Check for correct resource prefixes
+   - Ensure AWS Organizations access is properly configured
+
+2. **Permission Errors**
+   - Review IAM role policies in both management and linked accounts
+   - Verify service enablement in AWS Organizations
+   - Check for any service quotas or limits
+
+3. **Data Collection Failures**
+   - Verify S3 bucket permissions
+   - Check CloudWatch Logs for execution errors
+   - Ensure correct role ARNs are being used
+
+## Contributing
+
+Contributions to improve the solution are welcome! Please:
+
+1. Fork the repository
+2. Create a feature branch
+3. Submit a pull request with detailed description of changes
+4. Ensure all existing tests pass
+5. Add new tests as needed

data-collection/deploy/terraform/data-collection/main.tf

@@ -0,0 +1,50 @@
+data "aws_region" "current" {}
+data "aws_caller_identity" "current" {}
+data "aws_partition" "current" {}
+
+locals {
+  template_url = "https://aws-managed-cost-intelligence-dashboards-${data.aws_region.current.name}.s3.amazonaws.com/cfn/${var.tag_version}/data-collection/deploy-data-collection.yaml"
+}
+
+resource "aws_cloudformation_stack" "data_collection" {
+  name = "${var.resource_prefix}data-collection"
+  capabilities = [
+    "CAPABILITY_NAMED_IAM",
+    "CAPABILITY_AUTO_EXPAND"
+  ]
+  template_url = local.template_url
+
+  parameters = {
+    ResourcePrefix     = var.resource_prefix
+    DatabaseName       = var.database_name
+    DestinationBucket = var.destination_bucket
+    ManagementAccountID = var.management_account_id
+    ManagementAccountRole = var.management_account_role
+    MultiAccountRoleName = var.multi_account_role_name
+    Schedule = var.schedule
+    ScheduleFrequent = var.schedule_frequent
+    CFNSourceBucket = var.cfn_source_bucket
+    RegionsInScope = var.regions_in_scope
+    DatabaseName = var.database_name
+    DataBucketsKmsKeysArns = var.data_buckets_kms_keys_arns
+    IncludeTAModule   = var.include_ta_module ? "yes" : "no"
+    IncludeRDSUtilizationModule = var.include_rds_usage_module ? "yes" : "no"
+    IncludeOrgDataModule = var.include_org_data_module ? "yes" : "no"
+    IncludeRightsizingModule = var.include_ce_rightsizing_module ? "yes" : "no"
+    IncludeCostAnomalyModule = var.include_cost_anomaly_module ? "yes" : "no"
+    IncludeSupportCasesModule = var.include_support_cases_module ? "yes" : "no"
+    IncludeBackupModule = var.include_backup_module ? "yes" : "no"
+    IncludeInventoryCollectorModule = var.include_inventory_module ? "yes" : "no"
+    IncludeComputeOptimizerModule = var.include_compute_optimizer_module ? "yes" : "no"
+    IncludeECSChargebackModule = var.include_ecs_chargeback_module ? "yes" : "no"
+    IncludeBudgetsModule = var.include_budgets_module ? "yes" : "no"
+    IncludeTransitGatewayModule = var.include_transit_gateway_module ? "yes" : "no"
+    IncludeAWSFeedsModule = var.include_aws_feeds_module ? "yes" : "no"
+    IncludeHealthEventsModule = var.include_health_events_module ? "yes" : "no"
+    IncludeLicenseManagerModule = var.include_license_manager_module ? "yes" : "no"
+    IncludeServiceQuotasModule = var.include_service_quotas_module ? "yes" : "no"
+    IncludeQuickSightModule = var.include_quicksight_module ? "yes" : "no"
+  }
+
+  tags = var.tags
+}

data-collection/deploy/terraform/data-collection/outputs.tf

@@ -0,0 +1,9 @@
+output "stack_id" {
+  description = "The unique identifier for the stack"
+  value       = aws_cloudformation_stack.data_collection.id
+}
+
+output "stack_outputs" {
+  description = "Map of outputs from the CloudFormation stack"
+  value       = aws_cloudformation_stack.data_collection.outputs
+}

data-collection/deploy/terraform/data-collection/terraform.tfvars

@@ -0,0 +1,32 @@
+# Default values for variables
+resource_prefix = "CID-DC-"
+tag_version = "3.6.2"
+
+destination_bucket = "cid-data-"
+management_account_id = "987654321987"  # Required: Add your management account IDs as a comma separated list
+management_account_role = "Lambda-Assume-Role-Management-Account"  # Required: Add your management account role
+multi_account_role_name = "Optimization-Data-Multi-Account-Role"  # Required: Add your multi-account role name
+regions_in_scope = "us-east-1,eu-west-1,eu-central-1"  # Update with your desired regions
+data_buckets_kms_keys_arns = "KMS_KEY_ARN"
+schedule = "rate(1 day)"  # Update with your desired schedule
+schedule_frequent = "rate(1 day)"  # Update with your desired frequent schedule
+database_name = "optimization_data"
+cfn_source_bucket = "aws-managed-cost-intelligence-dashboards" # Don't Change
+include_ta_module = "true"
+include_rds_usage_module = "true"
+include_org_data_module = "true"
+include_ce_rightsizing_module = "true"
+include_cost_anomaly_module = "true"
+include_support_cases_module = "true"
+include_backup_module = "true"
+include_inventory_module = "true"
+include_pricing_module = "true"
+include_compute_optimizer_module = "false"
+include_ecs_chargeback_module = "true"
+include_budgets_module = "true"
+include_transit_gateway_module = "true"
+include_aws_feeds_module = "true"
+include_health_events_module = "true"
+include_license_manager_module = "true"
+include_service_quotas_module = "true"
+include_quicksight_module = "true"

data-collection/deploy/terraform/data-collection/variables.tf

@@ -0,0 +1,187 @@
+variable "resource_prefix" {
+  description = "Prefix to be used for all resources created by this module"
+  type        = string
+  default     = "CID-DC-"
+}
+
+variable "cfn_source_bucket" {
+  description = "Name of the S3 bucket where CloudFormation templates are stored"
+  type        = string
+  default     = "aws-managed-cost-intelligence-dashboards"
+}
+
+variable "destination_bucket" {
+  description = "Name of the S3 bucket where data will be stored"
+  type        = string
+  default     = "cid-data-"
+}
+
+variable "management_account_id" {
+  description = "Comma-delimited list of Account IDs for Management Account IDs"
+  type        = string
+}
+
+variable "management_account_role" {
+  description = "Management account role"
+  type        = string
+  default     = "Lambda-Assume-Role-Management-Account"
+}
+
+variable "multi_account_role_name" {
+  description = "Multi Account Role Name"
+  type        = string
+  default     = "Optimization-Data-Multi-Account-Role"
+}
+
+variable "regions_in_scope" {
+  description = "Comma-delimited list of AWS regions for resource data collection (e.g., \"us-east-1,eu-west-1,ap-northeast-1\")"
+  type        = string
+}
+
+variable "data_buckets_kms_keys_arns" {
+  description = "ARNs of KMS Keys for data buckets and/or Glue Catalog. Comma separated list, no spaces. Keep empty if data Buckets and Glue Catalog are not Encrypted with KMS. You can also set it to '*' to grant decrypt permission for all the keys."
+  type        = string
+}
+
+variable "schedule" {
+  description = "Cron expression for execution schedule.\")"
+  type        = string
+  default     = "rate(14 days)"
+}
+
+variable "schedule_frequent" {
+  description = "Cron expression for more frequent executions.\")"
+  type        = string
+  default     = "rate(1 day)"
+}
+
+variable "database_name" {
+  description = "Name of the Glue Database to be created"
+  type        = string
+  default     = "optimization_data"
+}
+
+variable "include_ta_module" {
+  description = "Whether to include the Trusted Advisor module"
+  type        = bool
+  default     = true
+}
+
+variable "include_rds_usage_module" {
+  description = "Whether to include the RDS Usage module"
+  type        = bool
+  default     = true
+}
+
+variable "include_org_data_module" {
+  description = "Whether to include the Organization Data module"
+  type        = bool
+  default     = true
+}
+
+variable "include_ce_rightsizing_module" {
+  description = "Whether to include the Cost Explorer Rightsizing module"
+  type        = bool
+  default     = true
+}
+
+variable "include_cost_anomaly_module" {
+  description = "Whether to include the Cost Anomaly module"
+  type        = bool
+  default     = true
+}
+
+variable "include_support_cases_module" {
+  description = "Whether to include the Support Cases module"
+  type        = bool
+  default     = true
+}
+
+variable "include_backup_module" {
+  description = "Whether to include the Backup module"
+  type        = bool
+  default     = true
+}
+
+variable "include_inventory_module" {
+  description = "Whether to include the Inventory module"
+  type        = bool
+  default     = true
+}
+
+variable "include_pricing_module" {
+  description = "Whether to include the Pricing module"
+  type        = bool
+  default     = true
+}
+
+variable "include_compute_optimizer_module" {
+  description = "Whether to include the Compute Optimizer module"
+  type        = bool
+  default     = true
+}
+
+variable "include_ecs_chargeback_module" {
+  description = "Whether to include the ECS Chargeback module"
+  type        = bool
+  default     = true
+}
+
+variable "include_budgets_module" {
+  description = "Whether to include the Budgets module"
+  type        = bool
+  default     = true
+}
+
+variable "include_transit_gateway_module" {
+  description = "Whether to include the Transit Gateway module"
+  type        = bool
+  default     = true
+}
+
+variable "include_aws_feeds_module" {
+  description = "Whether to include the AWS Feeds module"
+  type        = bool
+  default     = true
+}
+
+variable "include_health_events_module" {
+  description = "Whether to include the Health Events module"
+  type        = bool
+  default     = true
+}
+
+variable "include_license_manager_module" {
+  description = "Whether to include the License Manager module"
+  type        = bool
+  default     = true
+}
+
+variable "include_service_quotas_module" {
+  description = "Whether to include the Service Quotas module"
+  type        = bool
+  default     = true
+}
+
+variable "include_quicksight_module" {
+  description = "Whether to include the QuickSight module"
+  type        = bool
+  default     = true
+}
+
+variable "tags" {
+  description = "Tags to be added to all resources"
+  type        = map(string)
+  default     = {}
+}
+
+variable "tag_version" {
+  description = "GitHub tag version using for the deployment (e.g. 4.0.7)"
+  type        = string
+  default     = ""
+}
+
+validation {
+  condition     = var.tag_version == "" || can(regex("^\\d+\\.\\d+\\.\\d+$", var.tag_version))
+  error_message = "The tag_version must be in the format X.Y.Z where X, Y, and Z are digits (e.g., 4.0.7)"
+}