add initial ambient mc perf doc

[therealmitchconnors]

Description

Document the performance characteristics of Ambient Multi Cluster in preparation for Beta.

Reviewers

• Ambient
• Docs
• Installation
• Networking
• Performance and Scalability
• Extensions and Telemetry
• Security
• Test and Release
• User Experience
• Developer Infrastructure
• Localization/Translation

[keithmattix]

Needs one sentence of explanation, but otherwise LGTM.

[keithmattix]

Suggested change

As documented [here](/docs/ops/deployment/performance-and-scalability), the Istio control plane generally scales as the product of deployment changes, configuration changes, and the number of connected proxies. Ambient Multi Cluster adds two new dimensions to the Control Plane scalability story: number of remote clusters, and number of remote services. This means that adding 10 remote services to the mesh has substantially lower impact on the control plane performance than adding 10 local services.

As documented [here](/docs/ops/deployment/performance-and-scalability), the Istio control plane generally scales as the product of deployment changes, configuration changes, and the number of connected proxies. Ambient multicluster adds two new dimensions to the control plane scalability story: number of remote clusters, and number of remote services. Because the control plane is not programming proxies for remote clusters (assuming a multi-primary deployment topology), adding 10 remote services to the mesh has substantially lower impact on the control plane performance than adding 10 local services.

[howardjohn]

Its not the double encryption causing 346% increase though, right? At minimum its +1 TCP proxy hop, which is (likely) more expensive than the double TLS.

Is this also tested on a real cross-zone/cross-VPC/etc cloud? Or is the network path ~zero cost in the test? If not, that would be a major factor here as well

[keithmattix]

I believe the tests have no network cost since they're run with kind /cc @Stevenjin8

[Stevenjin8]

the +346% is due to having to reestablish an inner hbone for every connection. These tests were all run in kind locally.

[Stevenjin8]

so yeah, its not due to double encryption alone

[Stevenjin8]

Also, double encryption does not require double handshakes, we/I were just a bit lazy in our implementation. But we can also make the point that we can speed this up in the future to be roughly on par with request/response numbers

[howardjohn]

Ah right, forgot the 346 is for CRR not RR. RR is the more important number anyways

[Stevenjin8]

we should standardize on "multi cluster" vs "multicluster"

[dhawton]

Multicluster as a single word is the standardized term

[dhawton]

Some nits for consistency and conforming to the general casing used across the docs.

[dhawton]

Suggested change

This double-encryption does carry some overhead, however. The Data Plane Load test measures the response latency of traffic between pods in the same cluster, versus those in two different clusters, to understand the impact of double encryption on latency. Additionally, double encryption requires double handshakes, which disproportionately affects the latency of new connections to the remote cluster. As you can see below, our initial connections observed an average of 2.2 milliseconds(346%) additional latency, while requests using existing connections observed an increase of 0.13 milliseconds(72%). While these numbers appear significant, it is expected that most multicluster traffic will cross availability zones or regions, and the observed increase in overhead latency will be minimal compared to the overall transit latency between data centers.

This double encryption does carry some overhead, however. The data plane load test measures the response latency of traffic between pods in the same cluster, versus those in two different clusters, to understand the impact of double encryption on latency. Additionally, double encryption requires double handshakes, which disproportionately affects the latency of new connections to the remote cluster. As you can see below, our initial connections observed an average of 2.2 milliseconds (346%) additional latency, while requests using existing connections observed an increase of 0.13 milliseconds (72%). While these numbers appear significant, it is expected that most multicluster traffic will cross availability zones or regions, and the observed increase in overhead latency will be minimal compared to the overall transit latency between data centers.

[dhawton]

Suggested change

Our Multicluster Control Plane Load test created 300 services with 4000 endpoints in each of 10 clusters, and added these clusters to the mesh one at a time. The approximate control plane impact of adding a remote cluster at this scale was **1% of a CPU core, and 180 MB of memory**. At this scale, it should be safe to scale well beyond 10 clusters in a mesh with a properly scaled control plane. One item to note is that for Multicluster scalability, horizontally scaling the control plane will not help, as each control plane instance maintains a complete cache of remote services. Instead, we recommend modifying the resource requests and limits of the control plane to scale vertically to meet the needs of your multicluster mesh.

Our multicluster control plane load test created 300 services with 4000 endpoints in each of 10 clusters, and added these clusters to the mesh one at a time. The approximate control plane impact of adding a remote cluster at this scale was **1% of a CPU core, and 180 MB of memory**. At this scale, it should be safe to scale well beyond 10 clusters in a mesh with a properly scaled control plane. One item to note is that for multicluster scalability, horizontally scaling the control plane will not help, as each control plane instance maintains a complete cache of remote services. Instead, we recommend modifying the resource requests and limits of the control plane to scale vertically to meet the needs of your multicluster mesh.

[dhawton]

Suggested change

Multicluster deployments with Ambient mode enable you to offer truly globally resilient applications at scale with minimal overhead. In addition to its normal functions, the Istio control plane creates watches on all remote clusters to keep an up-to-date listing of what global services each cluster offers. The Istio dataplane can route traffic to these remote global services, either as a part of normal traffic distribution, or specifically when the local service is unavailable.

Multicluster deployments with ambient mode enable you to offer truly globally resilient applications at scale with minimal overhead. In addition to its normal functions, the Istio control plane creates watches on all remote clusters to keep an up-to-date listing of what global services each cluster offers. The Istio data plane can route traffic to these remote global services, either as a part of normal traffic distribution, or specifically when the local service is unavailable.