[STORY] As a SysAdmin of system using LLMs, I want actionable Reports on Red Teaming results of LLMs

[Cybonto]

UAT Specification

---
schemaVersion: "uat-v1.0"
issueID: ""
type: "story"
status: "pending-acceptance"
priority: 3
assignee: "Cybonto"

spec:
  userStory: "As a SysAdmin of system using LLMs, I want actionable Reports on Red Teaming results of LLMs."
  acceptanceCriteria:
    - given: "ViolentUTF is set up properly with Red Teaming datasets. "
      when: "ViolentUTF pipeline was executed and scorer results are ready."
      then: "A report is produced in pdf and json format."
    - given: "ViolentUTF past reports were generated saved."
      when: "I look up past reports from the Console."
      then: "I am able to open/download past reports"
    - given: "Scheduled report jobs are configured"
      when: "Reports are set to run weekly/monthly"
      then: "PDF reports are automatically generated and made available for download without manual intervention"
    - given: "Stakeholder email preferences are configured"
      when: "Scheduled reports complete successfully"
      then: "Stakeholders receive email notifications with download links for their reports"
  outOfScope:
    - "Real-time dashboards or live updating interfaces"
    - "New API endpoints beyond existing reports system"
    - "Custom database tables (use existing reports infrastructure)"
    - "WebSocket connections or real-time notifications"
  assumptions:
    - "ViolentUTF API reporting system is operational with PDF/HTML generation"
    - "Existing Celery background job system is working"
    - "Historical data collection has been running for baseline establishment"
    - "Email notification system is configured"

quality:
  performance:
    response_time_p95_ms: 500
    concurrent_attacks: 10
    max_payload_size_mb: 10
  security:
    authentication_required: true
    authorization_roles: ["security_engineer", "red_team"]
    audit_logging_required: true
  maintainability:
    code_coverage_min_percent: 80
    documentation_required: true
  accessibility:
    wcag_level: "AA"

relations:
  parent: "tba"
  dependsOn: [tba]
  relatesTo: [tba]
  blocks: [tba]

implementation:
  estimatedHours: 32  # Reduced due to reusing existing infrastructure
  requiredSkills:
    - "Python/FastAPI"
    - "Data Collection & Analysis"
    - "Report Template Design"
    - "Celery Background Tasks"
  testingStrategy:
    unit: true
    integration: true
    e2e: false  # Simpler workflow
    security: true
---

User Context

Limited Visibility into LLM Vulnerabilities: Without comprehensive red teaming reports, system admins lack visibility into potential vulnerabilities in the LLMs. This increases the risk of security breaches, data leaks, and the generation of harmful content.

Lack of Actionable Insights from Red Teaming: Red teaming exercises generate data, but system admins struggle to translate that data into concrete steps to improve LLM security and performance. They need clear reports that highlight vulnerabilities and suggest remediation strategies.

Compliance Concerns: Government agencies are subject to strict regulations regarding AI usage. System admins need to ensure that their LLM systems comply with these regulations, and red teaming reports can help demonstrate compliance.

Time-Consuming Manual Report Generation: Creating reports from red teaming results is a manual, tedious process. This takes time away from other critical tasks and can lead to delays in identifying and addressing vulnerabilities.

Difficulty in Tracking Historical Red Teaming Data:System admins need to track red teaming results over time to identify trends and measure the effectiveness of security improvements. The current system may lack robust historical data storage and retrieval capabilities.

Keeping up with Evolving Threats: The threat landscape for LLMs is constantly evolving. System admins need a red teaming solution that can adapt to new attack vectors and vulnerabilities.

UI/UX Mockups or API Design

Report Request Interface (using existing reports API):
{{tba}}

Report Types (extending existing system):
{{tba}}

Templates (HTML/PDF):
{{tba}}

API Changes

New endpoints: None (using existing reports system)

Modified endpoints: None

Deprecated endpoints: None

Breaking changes: None

Expected Outcomes

• Automated Generation of Actionable Red Teaming Reports:
  • The system automatically generates reports in PDF and JSON formats that summarize red teaming results and provide clear recommendations for improving LLM security and performance.
• Improved Visibility into LLM Vulnerabilities:
  • The reports provide system admins with a comprehensive view of potential vulnerabilities in the LLMs, allowing them to prioritize remediation efforts.
• Reduced Manual Effort:
  • Automation of report generation reduces the manual effort required, freeing up system admins to focus on other critical tasks.
• Enhanced Compliance:
  • The reports provide evidence of red teaming activities, which can help demonstrate compliance with government regulations and policies.
• Improved LLM Security Posture:
  • By identifying and addressing vulnerabilities, the system helps to improve the overall security posture of the LLMs.
• Data-Driven Decision Making:
  • The reports provide data-driven insights that can inform decision-making related to LLM security and performance.
• Proactive Threat Detection:
  • Red teaming exercises can help identify potential vulnerabilities before they are exploited by malicious actors.
• Scalable Solution:
  • The automated reporting system can scale to meet the evolving needs of the agency as its use of LLMs grows.
• Streamlined Workflow:
  • Integration with existing security tools and workflows streamlines the red teaming process and reduces the risk of errors.
• Faster Remediation:
  • The reports provide clear recommendations for remediation, allowing system admins to address vulnerabilities more quickly.

Success Criteria

• Report Generation Frequency: Reports are generated on a weekly/monthly basis as scheduled.
• Report Availability: Reports are readily available for download in PDF and JSON formats.
• Email Notifications: Stakeholders receive timely email notifications with download links for new reports.
• User Adoption: System admins and other stakeholders actively use the reports to improve LLM security and performance.
• Vulnerability Remediation: A measurable reduction in identified vulnerabilities over time as a result of the red teaming and reporting process. Tracked via subsequent red team exercises.
• Compliance Adherence: The red teaming process and reporting helps the agency demonstrate compliance with relevant regulations and policies.
• System Performance: The report generation process does not negatively impact the performance of the LLM systems.
• Stakeholder Satisfaction: Stakeholders are satisfied with the quality and usefulness of the reports. Measured via feedback surveys or interviews.
• Automation Efficiency: A significant reduction in the manual effort required for report generation.
• Data Accuracy: The reports accurately reflect the results of the red teaming exercises.

Primary Component

API Endpoints

Attack Framework

PyRIT

[Cybonto]

Additional things to consider:

• Cybersecurity focused stakeholders (ie. CISA)
• Compliance focused stakeholders (NIST, FedRAMP)
• Existing RT templates, Console reporting style and metrics.
• Common fields in security section of mainstream LLM model cards.

[Cybonto]

To make RT results actionable, the summary and key metrics should be meaningful to SOC, threat hunters, and auditors. Traditional Data Science metrics will not "cut it". We will also broaden the RT report scope to not just the LLMs but also the systems the LLMs run on, as interdependencies among AI-system components are high.

[Cybonto]

Top down design approach:
[ Stakeholder persona + workflows ]
↓
[ Role-based lenses ]
↓
[ Framework alignment ]
↓
[ Metric families ]
↓
[ Metric definitions ]
↓
[ Datasets & Data operations ]

[Cybonto]

A design document for the Red Teaming Reporting Structure and Metrics is being developed. This document presents a comprehensive design framework for an AI Security Dashboard/Report Feature tailored to serve government agency stakeholders of a Government AI-as-a-Service platforms. The dashboard addresses the unique challenges of AI security by integrating multiple trusted frameworks while providing role-specific views that enable effective decision-making.

Key Innovation: Stakeholder-Centric Design

Rather than building a generic security dashboard, this design is predicated on deep analysis of eight distinct stakeholder personas, from tactical operators like Developers and SOC Analysts to strategic decision-makers like CISOs and Authorizing Officials. Each persona operates under a "Triad of Pressures": compliance demands, oversight requirements, and evolving threat landscapes.

Core Design Principles

1. Stakeholder-Obsessed & Workflow-Driven: Every feature directly supports specific decisions and workflows
2. Common Data Core with Role-Based Lenses: Single source of truth presented through tailored interfaces
3. Grounded in Respected Frameworks: Built on NIST AI RMF, MITRE ATLAS™, and industry model card standards
4. Meaningful & Actionable Intelligence: Every metric drives specific actions, not just information display

Architectural Innovation: Three-Lens Approach

The dashboard implements three distinct viewing lenses:

• Tactical Lens: Granular, real-time data for hands-on operators (Developers, ISSOs, SOC Analysts)
• Managerial Lens: Aggregated trends and KPIs for program oversight (PMs, CORs)
• Strategic Lens: Executive summaries and risk roll-ups for leadership (CISOs, AOs)

Framework Integration Strategy

The design employs a hybrid framework approach that uniquely addresses AI security challenges:

• NIST AI RMF provides overarching risk management structure
• MITRE ATLAS delivers actionable threat intelligence for AI-specific attacks
• Model Card Standards establish quantitative safety benchmarks

This integration transforms isolated security findings into comprehensive risk assessments, enabling stakeholders to understand both the security and safety dimensions of AI risks.

Key Benefits

1. Unified Risk Picture: Bridges traditional divides between security, safety, and AI teams
2. Compliance Efficiency: Single exercises generate evidence across multiple requirements
3. Cross-Functional Synergies: SOC incidents automatically generate compliance evidence
4. Future-Ready: Adaptable framework accommodates emerging AI threats and regulations

Implementation Value

This dashboard design enables government agencies to:

• Make risk-based authorization decisions with complete confidence
• Maintain continuous ATO compliance with automated evidence gathering
• Respond to AI-specific threats with purpose-built detection and response capabilities
• Demonstrate contractor accountability through objective performance metrics
• Translate technical AI risks into mission impact for executive decision-making

A link to the first draft will be emailed.