Using `NatsRequestReplyMode.Direct` screws up OpenTelemetry traces

[aradalvand]

Observed behavior

Using NatsRequestReplyMode.Direct screws up traces.

var natsClient = new NatsClient(opts: new()
{
	Name = "NATS Client",
	Url = primarySettings!.NatsAddress.ToString(),
	RequestReplyMode = NatsRequestReplyMode.Direct,
});

Expected behavior

My suspicion is that it might be due to the following:

nats.net/src/NATS.Client.Core/NatsConnection.RequestReply.cs

Lines 43 to 59 in f674fca

	if (Opts.RequestReplyMode == NatsRequestReplyMode.Direct)
	{
	using var rt = _replyTaskFactory.CreateReplyTask(replySerializer, replyOpts.Timeout);
	requestSerializer ??= Opts.SerializerRegistry.GetSerializer<TRequest>();
	await PublishAsync(subject, data, headers, rt.Subject, requestSerializer, requestOpts, cancellationToken).ConfigureAwait(false);
	return await rt.GetResultAsync(cancellationToken).ConfigureAwait(false);
	}
	await using var sub1 = await CreateRequestSubAsync<TRequest, TReply>(subject, data, headers, requestSerializer, replySerializer, requestOpts, replyOpts, cancellationToken)
	.ConfigureAwait(false);
	await foreach (var msg in sub1.Msgs.ReadAllAsync(cancellationToken).ConfigureAwait(false))
	{
	return msg;
	}
	throw new NatsNoReplyException();

sub1.Msgs.ReadAllAsync (which is invoked when RequestReplyMode = SharedInbox) disposes activity instances after receiving a response:

nats.net/src/NATS.Client.Core/Internal/ActivityEndingMsgReader.cs

Lines 123 to 127 in f674fca

	while (_inner.TryRead(out var msg))
	{
	msg.Headers?.Activity?.Dispose();
	yield return msg;
	}

We might have to do something similar in the RequestReplyMode = Direct case?

Server and client version

Server: v2.11.0
Client: v2.6.0

Host environment

No response

Steps to reproduce

No response

[mtmk]

thanks @aradalvand this looks like a bug in direct mode. your assessment is spot on.