Add AWS Cost Explorer forecast automation

README.md

@@ -46,6 +46,19 @@ This foundational architecture is recommended for starting and allows deployment
 ## Architecture of Advanced Dashboards
 ![Advanced Architecture](assets/images/advanced-architecture.png  "Foundational Architecture")
 
+### Cost Explorer Forecast Integration
+
+The framework includes a dashboard for visualizing AWS Cost Explorer forecast data:
+
+- **Cost Visualization**: Visualize AWS Cost Explorer forecasts with confidence intervals
+- **Multi-dimension Analysis**: Analyze forecasts by service, account, region, and other dimensions
+- **Dashboard Integration**: Integrates with other CUDOS dashboards to provide comprehensive cost insights
+- **Multiple Metrics**: Support for different cost metrics (Unblended, Amortized, etc.)
+
+The dashboard is available as part of this repository, but **data collection** must be implemented separately through the [Cloud Intelligence Dashboards Data Collection](https://github.com/aws-solutions-library-samples/cloud-intelligence-dashboards-data-collection/) repository.
+
+For dashboard deployment details, see [Cost Forecast Dashboard](docs/cost_forecast.md).
+
 1. [AWS Data Exports](https://aws.amazon.com/aws-cost-management/aws-data-exports/) delivers daily the Cost & Usage Report (CUR2) to an [Amazon S3 Bucket](https://aws.amazon.com/s3/) in the Management Account.
 2. [Amazon S3](https://aws.amazon.com/s3/) replication rule copies Export data to a dedicated Data Collection Account S3 bucket automatically.
 3. [Amazon Athena](https://aws.amazon.com/athena/) allows querying data directly from the S3 bucket using an [AWS Glue](https://aws.amazon.com/glue/) table schema definition.
@@ -156,4 +169,4 @@ See [CONTRIBUTING](CONTRIBUTING.md#security-issue-notifications) for more inform
 This library is licensed under the MIT-0 License. See the [LICENSE](LICENSE) file.
 
 ## Notices
-Dashboards and their content: (a) are for informational purposes only, (b) represents current AWS product offerings and practices, which are subject to change without notice, and (c) does not create any commitments or assurances from AWS and its affiliates, suppliers or licensors. AWS content, products or services are provided “as is” without warranties, representations, or conditions of any kind, whether express or implied. The responsibilities and liabilities of AWS to its customers are controlled by AWS agreements, and this document is not part of, nor does it modify, any agreement between AWS and its customers.
+Dashboards and their content: (a) are for informational purposes only, (b) represents current AWS product offerings and practices, which are subject to change without notice, and (c) does not create any commitments or assurances from AWS and its affiliates, suppliers or licensors. AWS content, products or services are provided “as is” without warranties, representations, or conditions of any kind, whether express or implied. The responsibilities and liabilities of AWS to its customers are controlled by AWS agreements, and this document is not part of, nor does it modify, any agreement between AWS and its customers.

cfn-templates/cost-forecast-integration.yml

@@ -0,0 +1,73 @@
+AWSTemplateFormatVersion: '2010-09-09'
+Description: 'Integration patch for AWS Cost Forecast Dashboard in main CUDOS CloudFormation template'
+
+# This file contains the changes needed to integrate the Cost Forecast Dashboard into the main CFN template
+
+## 1. Add to Parameters section:
+  DeployCostForecastDashboard:
+    Type: String
+    Description: Deploy Cost Explorer Forecast Dashboard
+    Default: "no"
+    AllowedValues: ["yes", "no"]
+
+## 2. Add to ParameterLabels section:
+  DeployCostForecastDashboard:
+    default: "Deploy Cost Forecast Dashboard"
+
+## 3. Add to Label group in AWS::CloudFormation::Interface > ParameterGroups:
+# Add to the following parameter group:
+# - Label:
+#     default: CUDOS, Cost-Intelligence-Dashboard and KPI-Dashboard. Require deployment of CUR via CloudFormation (cur-aggregation.yaml) or manually (Dashboard data will appear within 24h after CUR creation).
+#   Parameters:
+#     - CURVersion
+#     - DeployCUDOSv5
+#     - DeployCostIntelligenceDashboard
+#     - DeployKPIDashboard
+#     - DeployCostForecastDashboard  <-- Add this line
+
+## 4. Add to Conditions section:
+  NeedCostForecastDashboard: !Equals [ !Ref DeployCostForecastDashboard, "yes" ]
+
+  # Also add to UseCUR2:
+  # UseCUR2:
+  #   Fn::And:
+  #     - !Equals [!Ref CURVersion, '2.0']
+  #     - Fn::Or:
+  #       - !Equals [ !Ref DeployCUDOSDashboard, "yes" ]
+  #       - !Equals [ !Ref DeployCUDOSv5, "yes" ]
+  #       - !Equals [ !Ref DeployCostIntelligenceDashboard, "yes" ]
+  #       - !Equals [ !Ref DeployKPIDashboard, "yes" ]
+  #       - !Equals [ !Ref DeployCostForecastDashboard, "yes" ] <-- Add this line
+
+## 5. Add to Resources section:
+  CostForecastDashboard:
+    Type: Custom::CidDashboard
+    Condition: NeedCostForecastDashboard
+    DependsOn:
+      - Setup
+    Properties:
+      Name: !Sub 'CostForecastDashboard${Suffix}'
+      ServiceToken: !GetAtt CidExec.Arn
+      Dashboard:
+        dashboard-id: cost-forecast-dashboard
+        resources: !If [IsChinaOrGovCloudRegion, 'https://aws-managed-cost-intelligence-dashboards.s3.amazonaws.com/hub/cost-forecast/resources.yaml', !Ref 'AWS::NoValue']
+      Tags:
+        - Key: IgnoreDependencies
+          Value: !If [NeedCostIntelligenceDashboard, !Ref CostIntelligenceDashboard, '']
+
+## 6. Add to Outputs section:
+  CostForecastDashboardURL:
+    Description: "URL of Cost Forecast Dashboard"
+    Condition: NeedCostForecastDashboard
+    Value: !GetAtt CostForecastDashboard.DashboardURL
+
+## 7. Add to NeedLakeFormationEnabled condition:
+  # NeedLakeFormationEnabled:
+  #   Fn::And:
+  #     - !Equals [ !Ref LakeFormationEnabled, "yes" ]
+  #     - Fn::Or:
+  #       - !Equals [ !Ref DeployCUDOSDashboard, "yes" ]
+  #       - !Equals [ !Ref DeployCUDOSv5, "yes" ]
+  #       - !Equals [ !Ref DeployCostIntelligenceDashboard, "yes" ]
+  #       - !Equals [ !Ref DeployKPIDashboard, "yes" ]
+  #       - !Equals [ !Ref DeployCostForecastDashboard, "yes" ] <-- Add this line

cid/builtin/core/__init__.py

@@ -1 +1,33 @@
 # This plugin implements Core dashboards
+
+def provides():
+    return {
+        'views': {
+            'account_map': {
+                'name': 'Account Map',
+                'providedBy': 'cid.builtin.core',
+                'File': 'queries/shared/account_map.sql',
+            },
+            'cost_forecast': {
+                'name': 'Cost Forecast View',
+                'providedBy': 'cid.builtin.core',
+                'File': 'queries/forecast/cost_forecast_view.sql',
+                'parameters': {
+                    'forecast_table_name': {
+                        'description': 'Name of the table containing forecast data',
+                        'default': 'cost_forecast_data'
+                    }
+                }
+            },
+        },
+        'datasets': {
+            'cost_forecast_dataset': {
+                'name': 'Cost Forecast Dataset',
+                'providedBy': 'cid.builtin.core',
+                'File': 'datasets/forecast/cost_forecast_dataset.json',
+                'dependsOn': {
+                    'views': ['cost_forecast']
+                }
+            }
+        }
+    }

cid/builtin/core/data/datasets/forecast/cost_forecast_dataset.json

@@ -0,0 +1,88 @@
+{
+  "DataSetId": "cost-forecast-dataset",
+  "Name": "Cost Forecast Dataset",
+  "PhysicalTableMap": {
+    "cost-forecast-table": {
+      "CustomSql": {
+        "DataSourceArn": "${athena_datasource_arn}",
+        "Name": "Cost Forecast",
+        "SqlQuery": "SELECT dimension, value, metric, start_date, end_date, mean_value, lower_bound, upper_bound, metric_name, dimension_name FROM ${athena_database_name}.cost_forecast",
+        "Columns": [
+          {
+            "Name": "dimension",
+            "Type": "STRING"
+          },
+          {
+            "Name": "value",
+            "Type": "STRING"
+          },
+          {
+            "Name": "metric",
+            "Type": "STRING"
+          },
+          {
+            "Name": "start_date",
+            "Type": "DATETIME"
+          },
+          {
+            "Name": "end_date",
+            "Type": "DATETIME"
+          },
+          {
+            "Name": "mean_value",
+            "Type": "DECIMAL"
+          },
+          {
+            "Name": "lower_bound",
+            "Type": "DECIMAL"
+          },
+          {
+            "Name": "upper_bound",
+            "Type": "DECIMAL"
+          },
+          {
+            "Name": "metric_name",
+            "Type": "STRING"
+          },
+          {
+            "Name": "dimension_name",
+            "Type": "STRING"
+          }
+        ]
+      }
+    }
+  },
+  "LogicalTableMap": {
+    "cost-forecast-logical-table": {
+      "Alias": "cost_forecast",
+      "DataTransforms": [
+        {
+          "CreateColumnsOperation": {
+            "Columns": [
+              {
+                "ColumnName": "forecast_range",
+                "ColumnId": "forecast_range",
+                "Expression": "ifelse(datediff(end_date, start_date) <= 1, 'Daily', 'Monthly')"
+              },
+              {
+                "ColumnName": "variance",
+                "ColumnId": "variance",
+                "Expression": "upper_bound - lower_bound"
+              },
+              {
+                "ColumnName": "variance_percentage",
+                "ColumnId": "variance_percentage",
+                "Expression": "ifelse(mean_value = 0, 0, (upper_bound - lower_bound) / mean_value * 100)"
+              }
+            ]
+          }
+        }
+      ],
+      "Source": {
+        "PhysicalTableId": "cost-forecast-table"
+      }
+    }
+  },
+  "ImportMode": "DIRECT_QUERY",
+  "Permissions": []
+}

cid/builtin/core/data/queries/forecast/cost_forecast_view.sql

@@ -0,0 +1,33 @@
+-- Create a view for cost forecast data
+CREATE OR REPLACE VIEW ${athena_database_name}.cost_forecast AS
+SELECT 
+    dimension,
+    value,
+    metric,
+    CAST(startdate AS date) AS start_date,
+    CAST(enddate AS date) AS end_date,
+    CAST(meanvalue AS decimal(18,4)) AS mean_value,
+    CAST(lowerbound AS decimal(18,4)) AS lower_bound,
+    CAST(upperbound AS decimal(18,4)) AS upper_bound,
+    CASE 
+        WHEN metric = 'UNBLENDED_COST' THEN 'Unblended Cost'
+        WHEN metric = 'BLENDED_COST' THEN 'Blended Cost'
+        WHEN metric = 'AMORTIZED_COST' THEN 'Amortized Cost'
+        WHEN metric = 'NET_UNBLENDED_COST' THEN 'Net Unblended Cost'
+        WHEN metric = 'NET_AMORTIZED_COST' THEN 'Net Amortized Cost'
+        WHEN metric = 'USAGE_QUANTITY' THEN 'Usage Quantity'
+        WHEN metric = 'NORMALIZED_USAGE_AMOUNT' THEN 'Normalized Usage'
+        ELSE metric
+    END AS metric_name,
+    CASE
+        WHEN dimension = 'SERVICE' THEN 'AWS Service'
+        WHEN dimension = 'LINKED_ACCOUNT' THEN 'Account ID'
+        WHEN dimension = 'LINKED_ACCOUNT_NAME' THEN 'Account Name'
+        WHEN dimension = 'REGION' THEN 'Region'
+        WHEN dimension = 'USAGE_TYPE' THEN 'Usage Type'
+        WHEN dimension = 'INSTANCE_TYPE' THEN 'Instance Type'
+        WHEN dimension = 'OPERATION' THEN 'Operation'
+        WHEN dimension = 'PURCHASE_TYPE' THEN 'Purchase Type'
+        ELSE dimension
+    END AS dimension_name
+FROM ${forecast_table_name}

cid/cli.py

@@ -288,6 +288,24 @@ def create_cur_proxy(ctx, cur_version, fields, **kwargs):
     ctx.obj.create_cur_proxy(cur_version=cur_version, fields=fields, **kwargs)
 
 
+@click.option('-v', '--verbose', count=True)
+@click.option('-y', '--yes', help='confirm all', is_flag=True, default=False)
+@cid_command
+def forecast(ctx, **kwargs):
+    """Generate AWS Cost Explorer forecasts
+
+    \b
+    Command options:
+     --time-period TEXT                    Time period for forecast (30, 90, 180, 365 days or custom)
+     --end-date TEXT                       End date for custom time period (YYYY-MM-DD)
+     --metrics TEXT                        Comma-separated list of metrics to forecast
+     --dimensions TEXT                     Comma-separated list of dimensions to forecast
+     --granularity TEXT                    Granularity of forecast (DAILY or MONTHLY)
+     --s3-bucket TEXT                      S3 bucket to upload results
+    """
+    ctx.obj.forecast(**kwargs)
+
+
 @click.option('-v', '--verbose', count=True)
 @click.option('-y', '--yes', help='confirm all', is_flag=True, default=False)
 @cid_command
@@ -302,5 +320,3 @@ def teardown(ctx, **kwargs):
 
 if __name__ == '__main__':
     main()
-
-

cid/commands/__init__.py

@@ -1,4 +1,3 @@
 # flake8: noqa: E401
 from cid.commands.init_qs import InitQsCommand
-
-
+from cid.commands.forecast import ForecastCommand