|
| 1 | +--- |
| 2 | +sidebar_label: Architecture |
| 3 | +--- |
| 4 | + |
| 5 | +# Architecture |
| 6 | + |
| 7 | +**VirtualMetric DataStream** is architected with enterprise security and data sovereignty as core principles. Unlike traditional solutions that require sending sensitive data to third-party cloud platforms for processing, **DataStream** keeps all your critical data within your environment while providing centralized management and visibility through a secure cloud control plane. |
| 8 | + |
| 9 | +<img src="/img/virtualmetric-datastream-architecture.png" alt="VirtualMetric DataStream - Enterprise Architecture Overview"/> |
| 10 | + |
| 11 | +## Security-First Architecture |
| 12 | + |
| 13 | +### Data Plane vs Control Plane Separation |
| 14 | + |
| 15 | +**DataStream** employs a strict separation between data plane and control plane operations, ensuring your sensitive security data never leaves your environment: |
| 16 | + |
| 17 | +**Control Plane (VirtualMetric Cloud)** |
| 18 | +- Multi-tenant SaaS platform for centralized management |
| 19 | +- Pipeline configuration and deployment |
| 20 | +- Fleet management across all Directors and Agents |
| 21 | +- Real-time statistics, monitoring, and alerting |
| 22 | +- Role-Based Access Control (RBAC) for team collaboration |
| 23 | +- Zero data processing or storage of customer logs |
| 24 | + |
| 25 | +**Data Plane (Customer Environment)** |
| 26 | +- All data processing occurs within customer-controlled infrastructure |
| 27 | +- Director processes, transforms, and routes data locally |
| 28 | +- Agents collect data and communicate directly with local Director |
| 29 | +- No customer data transmission to VirtualMetric Cloud |
| 30 | +- Complete data sovereignty and compliance control |
| 31 | + |
| 32 | +### Enterprise Security Benefits |
| 33 | + |
| 34 | +This architecture addresses critical enterprise security concerns: |
| 35 | + |
| 36 | +**Data Sovereignty** - All sensitive log data remains within your infrastructure, ensuring compliance with data residency requirements, industry regulations (GDPR, HIPAA, SOX), and corporate data governance policies. |
| 37 | + |
| 38 | +**Minimal Attack Surface** - Only a single HTTPS outbound connection required from Director to VirtualMetric Cloud for management, no inbound connections needed, and agents communicate exclusively with local Director infrastructure. |
| 39 | + |
| 40 | +**Zero Third-Party Data Exposure** - Raw log data never transmitted to external vendors, eliminates risks of data breaches during transit, and maintains complete control over sensitive security information. |
| 41 | + |
| 42 | +**Network Security** - Reduced firewall complexity with minimal external connections, no need to open firewall access from cloud to local systems, and simplified network security management. |
| 43 | + |
| 44 | +**Bandwidth Optimization** - Processes data locally before routing to destinations, eliminates unnecessary raw data transmission, and provides intelligent compression and filtering capabilities. |
| 45 | + |
| 46 | +## Deployment Architecture |
| 47 | + |
| 48 | +### VirtualMetric Cloud (Multi-Tenant) |
| 49 | + |
| 50 | +The centralized management platform provides: |
| 51 | +- **Management Interface** - Intuitive web-based console for pipeline configuration and monitoring |
| 52 | +- **Statistics & Analytics** - Real-time performance metrics, data flow visualization, and operational insights |
| 53 | +- **RBAC & Security** - Granular access controls, audit logging, and secure authentication |
| 54 | +- **Fleet Management** - Centralized deployment and configuration management across distributed environments |
| 55 | + |
| 56 | +### Customer Environment Options |
| 57 | + |
| 58 | +**VirtualMetric Director** can be deployed in multiple configurations to meet diverse enterprise requirements: |
| 59 | + |
| 60 | +#### Clustered Director (High Availability) |
| 61 | + |
| 62 | +- **Load Balancing** - Distributes processing load across multiple Director instances |
| 63 | +- **Automatic Failover** - Ensures continuous operation during maintenance or failures |
| 64 | +- **Scalable Processing** - Handles enterprise-scale data volumes with horizontal scaling |
| 65 | +- **Shared Configuration** - Synchronized pipeline configurations across cluster nodes |
| 66 | + |
| 67 | +#### Flexible Deployment Models |
| 68 | + |
| 69 | +**On-Premises Deployment** |
| 70 | +- Physical servers or virtual machines within customer data centers |
| 71 | +- Complete isolation from external networks if required |
| 72 | +- Integration with existing infrastructure and security controls |
| 73 | + |
| 74 | +**Cloud Deployment** |
| 75 | +- Customer-owned Azure, AWS, or other cloud environments |
| 76 | +- Maintains data sovereignty within customer cloud tenants |
| 77 | +- Leverages cloud-native services while preserving security isolation |
| 78 | + |
| 79 | +**Hybrid Deployment** |
| 80 | +- Directors in both on-premises and cloud environments |
| 81 | +- Unified management through single control plane |
| 82 | +- Flexible data routing based on location and requirements |
| 83 | + |
| 84 | +### Container and Serverless Support |
| 85 | + |
| 86 | +**Docker Containerization** |
| 87 | +- Lightweight, portable deployment across environments |
| 88 | +- Simplified installation and maintenance |
| 89 | +- Container orchestration support (Kubernetes, Docker Swarm) |
| 90 | +- Consistent runtime environment across platforms |
| 91 | + |
| 92 | +**Azure Serverless Integration** |
| 93 | +- **Director Proxy** - Azure Function-based secure data forwarding |
| 94 | +- Automatic scaling based on data volume |
| 95 | +- Pay-per-use cost optimization |
| 96 | + |
| 97 | +## Network Communication |
| 98 | + |
| 99 | +### Simplified Network Requirements |
| 100 | + |
| 101 | +**Outbound HTTPS (Director to VirtualMetric Cloud)** |
| 102 | +- Single port 443 connection for management communications |
| 103 | +- Control plane synchronization and configuration updates |
| 104 | +- Statistics reporting and health monitoring |
| 105 | +- Secure token-based authentication |
| 106 | + |
| 107 | +**Internal HTTPS (Agents to Director)** |
| 108 | +- Direct communication between Agents and local Director |
| 109 | +- No external connectivity required for Agents |
| 110 | +- Secure data transmission within customer environment |
| 111 | +- Simplified firewall configuration |
| 112 | + |
| 113 | +### Zero Inbound Connectivity |
| 114 | + |
| 115 | +**DataStream** requires no inbound connections from external networks, eliminating common security vulnerabilities: |
| 116 | +- No firewall rules for external access to internal systems |
| 117 | +- Reduced exposure to external threats |
| 118 | +- Simplified compliance and security auditing |
| 119 | +- Enhanced network security posture |
| 120 | + |
| 121 | +## Management Models |
| 122 | + |
| 123 | +### Managed (Default) |
| 124 | + |
| 125 | +- **Centralized Configuration** - Manage all pipelines through VirtualMetric portal |
| 126 | +- **Automatic Updates** - Seamless deployment of configuration changes and updates |
| 127 | +- **Real-Time Monitoring** - Comprehensive visibility across distributed infrastructure |
| 128 | +- **Collaborative Management** - Team-based access with RBAC controls |
| 129 | + |
| 130 | +### Self-Managed (Air-Gapped) |
| 131 | + |
| 132 | +For environments requiring complete network isolation: |
| 133 | + |
| 134 | +- **Offline Configuration** - Manual pipeline configuration and deployment |
| 135 | +- **Local Management** - Direct access to Director management interfaces |
| 136 | +- **Manual Updates** - Administrator-controlled updates and maintenance |
| 137 | +- **Complete Isolation** - Zero external connectivity requirements |
| 138 | + |
| 139 | +### Benefits for Enterprise Environments |
| 140 | + |
| 141 | +**Operational Efficiency** |
| 142 | +- Centralized management of distributed telemetry infrastructure |
| 143 | +- Reduced operational overhead with automated configuration deployment |
| 144 | +- Comprehensive monitoring and alerting capabilities |
| 145 | +- Streamlined troubleshooting and performance optimization |
| 146 | + |
| 147 | +**Security & Compliance** |
| 148 | +- Data never leaves customer-controlled environment |
| 149 | +- Simplified compliance with industry regulations |
| 150 | +- Reduced risk of data breaches and unauthorized access |
| 151 | +- Enhanced audit trail and governance capabilities |
| 152 | + |
| 153 | +**Scalability & Performance** |
| 154 | +- High availability and clustering support for mission-critical environments |
| 155 | +- Horizontal scaling to handle growing data volumes |
| 156 | +- Intelligent load balancing and resource optimization |
| 157 | +- Container and serverless deployment flexibility |
| 158 | + |
| 159 | +**Cost Optimization** |
| 160 | +- Eliminates costs associated with third-party data processing |
| 161 | +- Reduces bandwidth requirements through local processing |
| 162 | +- Flexible deployment models to optimize infrastructure costs |
| 163 | +- Pay-per-use serverless options for variable workloads |
0 commit comments