Support for load balancing across multiple external Elasticsearch nodes in ClusterLogForwarder

[hikarubi]

Description:
I'm implementing the logging operator with ClusterLogForwarder and need to send logs to multiple external Elasticsearch nodes with different IPs. I'm looking for the best approach to handle load balancing and failover.
Current Situation

• Multiple external Elasticsearch nodes, each with different IPs
• Need load balancing to distribute logs across these nodes
• Concerned about error handling when a node goes down

Attempted Solutions
Option 1: Multiple Outputs
Creating separate outputs for each Elasticsearch node works, but if a node goes down, it triggers numerous errors. But we don't want to send the same data to multiple nodes and create the same data x number_nodes.

Option 2: Service without Selector + Endpoints
I considered creating a Kubernetes Service without a selector and an Endpoints resource containing all Elasticsearch node IPs. However, this approach has SSL/TLS verification challenges.

Challenges Identified

SSL/TLS Verification: When using a Service for external Elasticsearch clusters, SSL verification fails unless the Elasticsearch certificates include the service name (my-elasticsearch.my-namespace.svc.cluster.local) in the SAN field.

Load Balancing Limitations: While Vector (the underlying forwarder) supports load balancing, the logging operator doesn't currently expose this functionality.

Questions

• Is there a recommended pattern for load balancing to multiple external Elasticsearch nodes?
• Would it be possible to expose Vector's load balancing capabilities through the ClusterLogForwarder API?
• Are there plans to support round-robin or other load balancing strategies for external outputs?