Add a test suite for the observability machinery

[aradalvand]

Proposed change

Potentially add a test project that performs some testing against activity/span creation, starting, stopping, attributes, etc.

Use case

Prevents regressions from surfacing over time, since this is a subtle area where emerging bugs won't be immediately obvious. See #911 (comment)

Contribution

No response

[mtmk]

we have a separate test project just for this but I can imagine running a single process might not be ideal since telemetry stuff can be process wide. any ideas how this can be done? e.g. should we spin a process per test?

[aradalvand]

I'm not sure we'll need a multi-process setup; my idea was along the lines of having one test case for every operation that we know should/could create a span, and then just using an activity listener to record some state in the test method and subsequently use that state to make some assertions.

Actually, kind of similar to what you already have in OpenTelemetryTest.cs (I hadn't noticed that before) — do you think that approach is inadequate somehow?

[mtmk]

I'm not sure we'll need a multi-process setup; my idea was along the lines of having one test case for every operation that we know should/could create a span, and then just using an activity listener to record some state in the test method and subsequently use that state to make some assertions.

Actually, kind of similar to what you already have in OpenTelemetryTest.cs (I hadn't noticed that before) — do you think that approach is inadequate somehow?

I think it works, but I'm not sure how otel stuff works under the bonnet. felt like a lot of global state being carried around but if you think the approach in OpenTelemetryTest.cs is good enough then I'm happy to carry on adding tests there.