[Beats receivers] Add exporter translation support for all currently unsupported Elasticsearch output features

[cmacknz]

• Relates Fall back to process runtime if otel runtime is unsupported elastic-agent#10087

The list of Elasticsearch output parameters that currently cannot be translated to the Elasticsearch exporter is currently defined below. These parameters will cause Elastic Agent to fall back to the sub-process runtime when they are used, preventing adoption of Beats receivers.

beats/libbeat/otelbeat/oteltranslate/outputs/elasticsearch/config_otel.go

Lines 166 to 187 in 4ba000d

	func checkUnsupportedConfig(cfg *config.C, logger *logp.Logger) error {
	if cfg.HasField("indices") {
	return fmt.Errorf("indices is currently not supported: %w", errors.ErrUnsupported)
	} else if cfg.HasField("pipelines") {
	return fmt.Errorf("pipelines is currently not supported: %w", errors.ErrUnsupported)
	} else if cfg.HasField("parameters") {
	return fmt.Errorf("parameters is currently not supported: %w", errors.ErrUnsupported)
	} else if value, err := cfg.Bool("allow_older_versions", -1); err == nil && !value {
	return fmt.Errorf("allow_older_versions:false is currently not supported: %w", errors.ErrUnsupported)
	} else if cfg.HasField("loadbalance") {
	return fmt.Errorf("loadbalance is currently not supported: %w", errors.ErrUnsupported)
	} else if cfg.HasField("non_indexable_policy") {
	return fmt.Errorf("non_indexable_policy is currently not supported: %w", errors.ErrUnsupported)
	} else if cfg.HasField("escape_html") {
	return fmt.Errorf("escape_html is currently not supported: %w", errors.ErrUnsupported)
	} else if cfg.HasField("kerberos") {
	return fmt.Errorf("kerberos is currently not supported: %w", errors.ErrUnsupported)
	}
	return nil
	}

Documented Elastic Agent Elasticsearch output parameters

• [ ] allow_older_versions we will always support connecting to older versions of ES. allow_older_version:false will not be supported

• [ ] pipelines // this setting does not work with standalone elastic-agent hence dropping support on beatreceivers

  • Standalone https://www.elastic.co/docs/reference/fleet/elasticsearch-output#output-elasticsearch-data-parsing-settings
  • Filebeat https://www.elastic.co/docs/reference/beats/filebeat/elasticsearch-output#pipelines-option-es

• kerberos

  • Completed by Add Kerberos support to beatsauth #47534 and [beatreceivers] Add support for kerberos elastic-agent#11163

• parameters

  • Standalone https://www.elastic.co/docs/reference/fleet/elasticsearch-output#output-elasticsearch-http-settings
  • Filebeat https://www.elastic.co/docs/reference/beats/filebeat/elasticsearch-output#_parameters

• [ ] escape_html Not planned to be done. Should be deprecated

Undocumented Elastic Agent Elasticsearch output parameters

• loadbalance

  • Filebeat https://www.elastic.co/docs/reference/beats/filebeat/elasticsearch-output#_loadbalance
  • This one is definitely used with some frequency.

• [ ] indices

  • Indices can be implemented using attribute processor in OTel

• [ ] non_indexable_policy is in tech preview and we will not support this on beatreceivers

  • Filebeat https://www.elastic.co/docs/reference/beats/filebeat/elasticsearch-output#_non_indexable_policy
  • Dead letter index should probably be deprecated, removed, and replaced with the failure store.

• ssl.restart_on_cert_change.*

  • Filebeat https://www.elastic.co/docs/reference/beats/filebeat/configuration-ssl#exit_on_cert_change_enabled

Undocumented Elastic Agent Elasticsearch output parameters

[elasticmachine]

Pinging @elastic/elastic-agent-data-plane (Team:Elastic-Agent-Data-Plane)

[khushijain21]

Kerberos settings cannot be directly implemented on ES exporter for reasons described below.

Kerberos authentication is performed at application level, but github.com/elastic/go-elasticsearch only accepts Transport in its constructor. However, http.Transport should not be used for authentication according to Golang documentation.Ref

We may have to support configuring kerberos authentication parameters on https://github.com/elastic/go-elasticsearch/ direclty

[khushijain21]

POC for Kerberos support on go-elasticsearch library and es-exporter

elastic/go-elasticsearch#1030

https://github.com/open-telemetry/opentelemetry-collector-contrib/compare/main...khushijain21:opentelemetry-collector-contrib:kerberos?expand=1

I have manually tested this and it works.

cc: @carsonip @lahsivjar

[axw]

Kerberos authentication is performed at application level, but github.com/elastic/go-elasticsearch only accepts Transport in its constructor. However, http.Transport should not be used for authentication according to Golang documentation.Ref

That's a bit of a misread IMO. It's OK to implement simple authentication this way, but not complex challenge/response authentication that involve multiple round trips.

opentelemetry-collector's auth framework relies on the RoundTripper interface for authentication: https://github.com/open-telemetry/opentelemetry-collector/blob/c480657097695250c5c0b5117fb32c9d2bb5a039/extension/extensionauth/client.go#L12-L18

RoundTrip should not modify the request, but it can intercept and pass through a modified clone of the request. That's what the existing collector auth extensions do.

I believe the "RoundTrip should not attempt to handle higher-level protocol details such as redirects, authentication, or cookies." bit is referring to more complex auth flows that involve making multiple requests, such as in NTLM. I see in elastic/go-elasticsearch#1030 that we are sort of doing that, but I'm wondering if that is necessary - it doesn't appear to be. You have some code that checks for a WWW-Authenticate: Negotiate response header and only then sends the Authorization request header. Can we unconditionally set the request header?

We may have to support configuring kerberos authentication parameters on https://github.com/elastic/go-elasticsearch/ direclty

Before we go adding Kerberos to go-elasticsearch directly, let's please look into building on the collector's auth framework. Ideally I'd like to pull the Kerberos support out of the OTel Collector Kafka components into an auth extension, and we can use it in both places.

[khushijain21]

You have some code that checks for a WWW-Authenticate: Negotiate response header and only then sends the Authorization request header. Can we unconditionally set the request header?

We should not - this is because a server could support multiple authentications like Kerberos, basic auth, API. If a client sends "WWW-Authenticate: Negotiate` unconditionally, that would mean server will "only" authenticate the client via Kerberos. A user cannot login via basic-auth in this case. And we also only set this header if a user configures Kerberos credentials and if the server supports it - otherwise continue with original http client.

1. When negotiate header is passed, server attempts kerberos authentication and fails

➜  libbeat git:(pipelines) ✗ curl -i --negotiate -u admin:testing -X GET "http://localhost:9203"
HTTP/1.1 401 Unauthorized
WWW-Authenticate: Basic realm="security" charset="UTF-8"
WWW-Authenticate: Negotiate
WWW-Authenticate: Bearer realm="security"
WWW-Authenticate: ApiKey
content-type: application/json;charset=utf-8
content-length: 483

{"error":{"root_cause":[{"type":"security_exception","reason":"missing authentication credentials for REST request [/]","header":{"WWW-Authenticate":["Basic realm=\"security\" charset=\"UTF-8\"","Negotiate","Bearer realm=\"security\"","ApiKey"]}}],"type":"security_exception","reason":"missing authentication credentials for REST request [/]","header":{"WWW-Authenticate":["Basic realm=\"security\" charset=\"UTF-8\"","Negotiate","Bearer realm=\"security\"","ApiKey"]}},"status":401}%    

2. Server successfully connects the client when negotiate header is not passed using basic auth

➜  libbeat git:(pipelines) ✗ curl -i -u admin:testing -X GET "http://localhost:9203" 
HTTP/1.1 200 OK
X-elastic-product: Elasticsearch
content-type: application/json
content-length: 547

{
  "name" : "51d8da2584b8",
  "cluster_name" : "docker-cluster",
  "cluster_uuid" : "OoqJGsNGRKOdt9Jrwr_JLQ",
  "version" : {
    "number" : "8.0.0-SNAPSHOT",
    "build_flavor" : "default",
    "build_type" : "docker",
    "build_hash" : "3ca7e2334a69f78fcc51612fdab9b7bd8d43b9d8",
    "build_date" : "2022-02-10T12:48:49.806743385Z",
    "build_snapshot" : true,
    "lucene_version" : "9.0.0",
    "minimum_wire_compatibility_version" : "7.17.0",
    "minimum_index_compatibility_version" : "7.0.0"
  },
  "tagline" : "You Know, for Search"
}

[khushijain21]

Kerberos authentication details are taken from this library https://github.com/jcmturner/gokrb5/blob/master/v8/spnego/http.go#L82-L123 here which also implements this on application level.

I believe the "RoundTrip should not attempt to handle higher-level protocol details such as redirects, authentication, or cookies." bit is referring to more complex auth flows that involve making multiple requests,

I agree, but in our case we are involving multiple requests so it still falls in the more complex auth flows

[khushijain21]

PR to support escape_html and loadbalance parameters on elasticsearchexporter

open-telemetry/opentelemetry-collector-contrib#43746
open-telemetry/opentelemetry-collector-contrib#43769

[axw]

We should not - this is because a server could support multiple authentications like Kerberos, basic auth, API. If a client sends "WWW-Authenticate: Negotiate` unconditionally, that would mean server will "only" authenticate the client via Kerberos. A user cannot login via basic-auth in this case. And we also only set this header if a user configures Kerberos credentials and if the server supports it - otherwise continue with original http client.

Sure, but why would you configure the collector with Kerberos then? Does this sort of behaviour exist in other auth methods? e.g. if you specify an API Key, and the server isn't configured to support API Key auth, will the client fall back to basic auth? Even if it does, it is important that it does?

The challenge-response approach is important for auth between clients (e.g. browsers) that can authenticate to servers with different mechanisms, but where the client doesn't know what the server supports ahead of time. Is that relevant for the collector/agent?

[axw]

PR to support escape_html

I've never understood why Beats even has this configuration. Just went and had a dig into the history, and it seems in the distant past (9 years ago!) that JSON encoding escaped HTML by default, and there was a request to disable that through the option added to Go 1.7: #2581

Can we drop this option, and just never escape HTML? It's not relevant in the context of encoding Elasticsearch documents.

[khushijain21]

Sure, but why would you configure the collector with Kerberos then?

I see your point now. In that case, we should be able to implement this on beatsauth extension and unconditonally set Negotiate header when kerberos is configured.

Edit: Updated PR elastic/opentelemetry-collector-components#828

[axw]

Thanks @khushijain21. That sounds good for the immediate future. In parallel I think we should propose a Kerberos auth extension upstream (in collector-contrib), so users who require Kerberos are not tied to EDOT.