feat(tracing): Port `sample_rand` to POTel #4106

sentrivana · 2025-02-27T13:13:30Z

Port sample_rand to potel-base. See spec.

There are now two places where a sample_rand might be generated:

If we're explicitly propagating with continue_trace, we'll backfill sample_rand on the propagation context like on master, either using the incoming one or generating a new one from the incoming sampled/sample_rate.
Otherwise, we generate a new sample_rand in the Sampler.

The generated sample_rand is then saved on the trace state.

This change fixes most of the failures in the Common test suite.

Closes #4027

codecov · 2025-02-27T13:14:55Z

❌ 108 Tests Failed:

Tests completed	Failed	Passed	Skipped
20866	108	20758	4476

View the top 3 failed test(s) by shortest run time

 tests.integrations.celery.integration_tests.test_celery_beat_cron_monitoring

Stack Traces | 0s run time

.../hostedtoolcache/Python/3.7.17.../x64/lib/python3.7/importlib/__init__.py:127: in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
<frozen importlib._bootstrap>:1006: in _gcd_import
    ???
<frozen importlib._bootstrap>:983: in _find_and_load
    ???
<frozen importlib._bootstrap>:953: in _find_and_load_unlocked
    ???
<frozen importlib._bootstrap>:219: in _call_with_frames_removed
    ???
<frozen importlib._bootstrap>:1006: in _gcd_import
    ???
<frozen importlib._bootstrap>:983: in _find_and_load
    ???
<frozen importlib._bootstrap>:967: in _find_and_load_unlocked
    ???
<frozen importlib._bootstrap>:677: in _load_unlocked
    ???
<frozen importlib._bootstrap_external>:728: in exec_module
    ???
<frozen importlib._bootstrap>:219: in _call_with_frames_removed
    ???
.../celery/integration_tests/__init__.py:7: in <module>
    from celery.beat import Scheduler
.tox/py3.7-celery-v5.0.5/lib/python3.7.../site-packages/celery/beat.py:22: in <module>
    from . import __version__, platforms, signals
.tox/py3.7-celery-v5.0.5/lib/python3.7.../site-packages/celery/signals.py:14: in <module>
    from .utils.dispatch import Signal
.tox/py3.7-celery-v5.0.5/lib/python3.7.../celery/utils/__init__.py:16: in <module>
    from .nodenames import nodename, nodesplit, worker_direct
.tox/py3.7-celery-v5.0.5/lib/python3.7.../celery/utils/nodenames.py:6: in <module>
    from kombu.entity import Exchange, Queue
.tox/py3.7-celery-v5.0.5/lib/python3.7.../site-packages/kombu/entity.py:7: in <module>
    from .serialization import prepare_accept_content
.tox/py3.7-celery-v5.0.5/lib/python3.7.../site-packages/kombu/serialization.py:440: in <module>
    for ep, args in entrypoints('kombu.serializers'):  # pragma: no cover
.tox/py3.7-celery-v5.0.5/lib/python3.7.../kombu/utils/compat.py:82: in entrypoints
    for ep in importlib_metadata.entry_points().get(namespace, [])
E   AttributeError: 'EntryPoints' object has no attribute 'get'

 tests.integrations.celery.test_celery

Stack Traces | 0s run time

ImportError while importing test module '.../integrations/celery/test_celery.py'.
Hint: make sure your test modules/packages have valid Python names.
Traceback:
.tox/py3.7-celery-v5.0.5/lib/python3.7.../site-packages/_pytest/python.py:617: in _importtestmodule
    mod = import_path(self.path, mode=importmode, root=self.config.rootpath)
.tox/py3.7-celery-v5.0.5/lib/python3.7.../site-packages/_pytest/pathlib.py:567: in import_path
    importlib.import_module(module_name)
.../hostedtoolcache/Python/3.7.17.../x64/lib/python3.7/importlib/__init__.py:127: in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
<frozen importlib._bootstrap>:1006: in _gcd_import
    ???
<frozen importlib._bootstrap>:983: in _find_and_load
    ???
<frozen importlib._bootstrap>:967: in _find_and_load_unlocked
    ???
<frozen importlib._bootstrap>:677: in _load_unlocked
    ???
.tox/py3.7-celery-v5.0.5/lib/python3.7.../_pytest/assertion/rewrite.py:186: in exec_module
    exec(co, module.__dict__)
.../integrations/celery/test_celery.py:6: in <module>
    from celery import Celery, VERSION
E   ImportError: cannot import name 'Celery' from 'celery' (.../sentry-python/sentry-python/.tox/py3.7-celery-v5.0.5/lib/python3.7.../site-packages/celery/__init__.py)

 tests.integrations.celery.test_update_celery_task_headers

Stack Traces | 0s run time

.../integrations/celery/beat.py:28: in <module>
    from celery import Task, Celery  # type: ignore
E   ImportError: cannot import name 'Task' from 'celery' (.../sentry-python/sentry-python/.tox/py3.7-celery-v5.0.5/lib/python3.7.../site-packages/celery/__init__.py)

During handling of the above exception, another exception occurred:
.../integrations/celery/test_update_celery_task_headers.py:7: in <module>
    from sentry_sdk.integrations.celery import _update_celery_task_headers
.../integrations/celery/__init__.py:9: in <module>
    from sentry_sdk.integrations.celery.beat import (
.../integrations/celery/beat.py:37: in <module>
    raise DidNotEnable("Celery not installed")
E   sentry_sdk.integrations.DidNotEnable: Celery not installed

To view more test analytics, go to the Test Analytics Dashboard
_{📋 Got 3 mins? Take this short survey to help us improve Test Analytics.}

tests/integrations/opentelemetry/test_propagator.py

Co-authored-by: Daniel Szoke <[email protected]>

antonpirker

Some questions have been answered off Github and all in all looks good.

The one comment from @szokeasaurusrex about rounding the sample_rand value before storing it trace_state could be addressed, but I guess it should work without it. But maybe better than safe then sorry.

sentrivana · 2025-03-26T15:16:04Z

That should hopefully be it for the functional changes -- adding test cases for the scenarios discussed in the reviews now.

szokeasaurusrex

Thanks for the updates, identified a few more suggestions but looks pretty good

szokeasaurusrex · 2025-03-27T11:52:23Z

sentry_sdk/integrations/opentelemetry/sampler.py

+        if parent_sample_rand is None:
+            return None
+
+        return _round_sample_rand(parent_sample_rand)


Maybe we should double-check with @cleptric and/or @jan-auer, but in order to maintain future-compatibility, I don't think we should be rounding the parent_sample_rand.

Suggested change

return _round_sample_rand(parent_sample_rand)

return Decimal(parent_sample_rand)

By rounding it, we potentially modify the DSC (which should be frozen) if the head SDK was rounding differently. This could become a problem if in the future, we were to change the format of the sample_rand value (e.g. by increasing/decreasing precision). If the head SDK had already been updated use the new format, but the user was still using an older Python SDK version, we'd be changing the value back to the old precision here, which we might not want to be doing.

szokeasaurusrex · 2025-03-27T11:56:28Z

sentry_sdk/integrations/opentelemetry/sampler.py

+    if sample_rand is not None:
+        trace_state = trace_state.update(
+            TRACESTATE_SAMPLE_RAND_KEY, f"{sample_rand:.6f}"  # noqa: E231
+        )
+


[optional] since it seems like we are setting the sample_rand both here and above in dropped_result using the same duplicated code, perhaps we can extract this code into a separate function? If we did this, we could also potentially more other stuff we set to that new function (this might be out of scope for this PR though)

If you think this change would be too much hassle, feel free to ignore this suggestion

moved here https://github.com/getsentry/sentry-python/pull/4106/files#diff-59aa7195d955e153b5cdd730f888994996a72eaf5e9ea174335ce961841584a9R154

szokeasaurusrex · 2025-03-27T11:59:55Z

sentry_sdk/integrations/opentelemetry/sampler.py

+            sample_rand = parent_sample_rand
+        else:
+            # We are the head SDK and we need to generate a new sample_rand
+            sample_rand = cast(Decimal, _generate_sample_rand(str(trace_id), (0, 1)))


This cast is quite confusing.

Could we please try to modify _generate_sample_rand's signature to return Decimal rather than Optional[Decimal]? That would eliminate the need for a cast

Suggested change

sample_rand = cast(Decimal, _generate_sample_rand(str(trace_id), (0, 1)))

sample_rand = _generate_sample_rand(str(trace_id), (0, 1))

szokeasaurusrex · 2025-03-27T12:02:37Z

sentry_sdk/tracing_utils.py

+    except Exception as ex:
+        logger.debug(f"Failed to round sample_rand {sample_rand}: {ex}")


What exception exactly can the above raise? Doesn't setting the context avoid the possibility of an InvalidOperation?

If we can get rid of the try/except, that would allow us to make the signature of this function and _generate_sample_rand a Decimal.

sentrivana · 2025-03-27T14:15:17Z

@szokeasaurusrex Here's a compact view of the changes addressing your last review: https://github.com/getsentry/sentry-python/pull/4106/files/9380196498cd4520b207957e69696410859a1844..185cc15e158ef0ff4c2e072e2038401e799fba37

szokeasaurusrex

Looks good!

szokeasaurusrex · 2025-03-27T14:45:32Z

sentry_sdk/integrations/opentelemetry/sampler.py

                attributes,
                sample_rate=sample_rate_to_propagate,
-                sample_rand=sample_rand,
+                sample_rand=None if sample_rand == parent_sample_rand else sample_rand,


[question] What is the purpose of comparing sample_rand to parent_sample_rand? We might want to clarify this with a code comment.

sampled_result (and dropped_result as well) will use the sample_rand provided to overwrite it in the trace state. Since we don't want to overwrite incoming parent_sample_rand (since we would potentially change the precision), we only pass it to sampled_result if it's different from parent_sample_rand -- i.e., we generated it ourselves.

Ngl I am still a bit confused. Didn't we get rid of the code that would potentially change the precision?

We got rid of the rounding code but not the precision code, we still format sample_rand to 6 decimal places here https://github.com/getsentry/sentry-python/pull/4106/files#diff-59aa7195d955e153b5cdd730f888994996a72eaf5e9ea174335ce961841584a9R169

In any case, the bigger picture here is just: if we have a halfway valid incoming sample_rand, don't tamper with it, even if it'd mean just adding a few zeroes at the end.

okay, fair enough

sentrivana commented Mar 20, 2025

View reviewed changes

tests/integrations/opentelemetry/test_propagator.py Show resolved Hide resolved

sentrivana added 9 commits March 21, 2025 09:10

.

e53c7ec

.

5f3051e

Read or generate sample_rand

d22d8a4

comment

65935f5

fix

bac69a7

.

b0e6d10

.

a482084

fix type

9b24fe5

Add sample_rand to propagator tests

49a3507

sentrivana force-pushed the ivana/potel/port-sample-rand branch from 04dc3b4 to 49a3507 Compare March 21, 2025 08:11

sentrivana added 10 commits March 21, 2025 10:37

Merge branch 'potel-base' into ivana/potel/port-sample-rand

3d4953e

facepalm

1332ebc

unused import

4fe4b2e

.

8d14de5

unused import

50db70f

patch correct func

67ca0e5

.

248306f

fix

ab3d868

Merge branch 'potel-base' into ivana/potel/port-sample-rand

d52b361

.

fc610e3

sentrivana commented Mar 24, 2025

View reviewed changes

tests/integrations/opentelemetry/test_propagator.py Outdated Show resolved Hide resolved

sentrivana added 7 commits March 24, 2025 12:42

.

541a83d

.

46e6f9b

precommit

b57ee5e

.

8d917f5

.

d2055d9

.

8523fde

.

948b926

sentrivana and others added 5 commits March 25, 2025 15:37

Update tests/integrations/opentelemetry/test_propagator.py

5db195a

Co-authored-by: Daniel Szoke <[email protected]>

Apply suggestions from code review

1b118cf

Co-authored-by: Daniel Szoke <[email protected]>

some things

8754c5e

add link

8830e34

simplify logic in sampler

6abb38c

sentrivana linked an issue Mar 26, 2025 that may be closed by this pull request

Port sample-rand to POTel #4027

Closed

antonpirker approved these changes Mar 26, 2025

View reviewed changes

move rounding to separate helper func

3246d38

sentrivana added 5 commits March 26, 2025 16:16

Merge branch 'potel-base' into ivana/potel/port-sample-rand

6ef0ebd

more lint

a0f1a35

tests

34cb097

.

be19ff4

fixes

9711c99

sentrivana requested a review from szokeasaurusrex March 27, 2025 09:15

sentrivana added 3 commits March 27, 2025 10:23

simplify this -- sample_rand will never be None here

bd6312f

Merge branch 'potel-base' into ivana/potel/port-sample-rand

d9fe5f9

Merge branch 'potel-base' into ivana/potel/port-sample-rand

9fe8606

szokeasaurusrex reviewed Mar 27, 2025

View reviewed changes

sentrivana added 4 commits March 27, 2025 14:03

Merge branch 'potel-base' into ivana/potel/port-sample-rand

9380196

dont round incoming sample_rand

c3d0156

misleading comments

c18cdf0

fix type signature

185cc15

sentrivana requested a review from szokeasaurusrex March 27, 2025 14:15

szokeasaurusrex approved these changes Mar 27, 2025

View reviewed changes

sentrivana merged commit fc8fa9f into potel-base Mar 27, 2025
106 of 128 checks passed

sentrivana deleted the ivana/potel/port-sample-rand branch March 27, 2025 14:58

sentrivana mentioned this pull request Mar 27, 2025

POtel implementation base branch (aka SDK 3.0.0) #3152

Closed

BYK mentioned this pull request Sep 3, 2025

breaking: Remove Spotlight Django integration #4747

Merged

	return _round_sample_rand(parent_sample_rand)
	return Decimal(parent_sample_rand)

	sample_rand = cast(Decimal, _generate_sample_rand(str(trace_id), (0, 1)))
	sample_rand = _generate_sample_rand(str(trace_id), (0, 1))

		except Exception as ex:
		logger.debug(f"Failed to round sample_rand {sample_rand}: {ex}")

feat(tracing): Port sample_rand to POTel #4106

feat(tracing): Port sample_rand to POTel #4106

Uh oh!

Conversation

sentrivana commented Feb 27, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

codecov bot commented Feb 27, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

❌ 108 Tests Failed:

Uh oh!

Uh oh!

Uh oh!

antonpirker left a comment

Choose a reason for hiding this comment

Uh oh!

sentrivana commented Mar 26, 2025

Uh oh!

szokeasaurusrex left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

sentrivana commented Mar 27, 2025

Uh oh!

szokeasaurusrex left a comment

Choose a reason for hiding this comment

Uh oh!

szokeasaurusrex Mar 27, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

sentrivana Mar 27, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

feat(tracing): Port `sample_rand` to POTel #4106

feat(tracing): Port `sample_rand` to POTel #4106

sentrivana commented Feb 27, 2025 •

edited

Loading

codecov bot commented Feb 27, 2025 •

edited

Loading

szokeasaurusrex Mar 27, 2025 •

edited

Loading

sentrivana Mar 27, 2025 •

edited

Loading