Rethink annotation design for better maintainability

[timoreimann]

CCM currently employs a set of annotations to control LBs. While these were easy to handle initially, nowadays they are becoming more and more challenging to maintain both for CCM developers and consumers; this is mostly due to the way they may (sometimes should) interact with each other. One example is how (secure) protocols affect specified ports and the corresponding validation.

We should think about alternative approaches to configuring LBs.

Versioning-wise, we should probably take the opportunity to introduce a breaking change as we promote CCM to v1 (which does not mean that we are going to deprecate the old, annotation-based approach for sure; more likely, we're going to continue supporting it but add new configuration features to v1 only).

[timoreimann]

Two ideas:

1. Consolidate annotations so that parameters that belong together must also be specified together. Example from @andykent in a different issue:

service.beta.kubernetes.io/do-loadbalancer-protocols: http:80, https:443, http2:8080

2. Define a custom CRD. This would allow for a well-structured configuration of parameters together with proper validation options. Contour took a similar route with their IngressRoute CRD, and so does Traefik.

[timoreimann]

Interesting note: Traefik went down a very similar route by starting off with simple annotations, gradually providing more consolidated alternatives, but eventually ending up embracing CRDs. To be fair, Traefik is an Ingress controller with far more options, so the challenge is not quite the same as an L3/L4 LB. Then again, DO LB's is somewhere in-between as it also provides L7 features (e.g., HTTP* support).

Personally, I am somewhat favoring the CRD approach as well, but I'm very open to discuss options.

[sjparkinson]

I've been trying out the configurations, hitting a few bumps as described, so thought it'd be worth weighing in. Also worth looking at how GKE ingress resources can be configured.

At this time DigitalOcean load balancers have ~7 settings to configure. The control panel does a great job of presenting and managing the various options for each of them, the most complex being the forwarding rules.

These settings are represented as ~24 annotations in the Kubernetes ingress service. I like the simplicity of annotations for the flag related settings, I've not had issues with any of those. The complexity comes with configuring forwarding rules using the current assortment of annotations.

I'd love to see something like the BackendConfig that you can define on a GKE ingress, or an annotation that allows you to define all forwarding rules in a single annotation.

service.beta.kubernetes.io/do-loadbalancer-forwarding-rules: |
  [
    { "protocol": "TCP", "port": 80 },
    { "protocol": "TCP", "port": 443, "tls-passthough": true },
  ]