feat(develop-docs): Add alternative masking strategies #6619
Conversation
Instructions and example for changelogPlease add an entry to Example: ## Unreleased
### Features
- Add alternative masking strategies ([#6619](https://github.com/getsentry/sentry-cocoa/pull/6619))If none of the above apply, you can opt out of this check by adding |
|
philprime
force-pushed
the
philprime/masking-dev-docs
branch
2 times, most recently
from
83b0f8c to
c887b99
Compare
philprime
force-pushed
the
philprime/masking-dev-docs
branch
from
c887b99 to
c73f441
Compare
|
Also consider these approaches:
|
philprime
requested review from
itaybre,
noahsmartin and
philipphofmann
as code owners
philipphofmann
left a comment
philipphofmann
left a comment
There was a problem hiding this comment.
Thanks for the great summary
|
|
||
| ### Goals | ||
|
|
||
| - Detect and mask sensitive regions without requiring app developer annotations. |
There was a problem hiding this comment.
l: It's unclear to me what app developer annotations are exactly.
There was a problem hiding this comment.
q: Would developer annotations help deterministically mask SwiftUI views? If that's the case, would a hybrid approach using viewhierarchy for UIKit and dev annotations for swift UI be acceptable?
|
|
||
| - No private APIs in production App Store builds. | ||
| - No dependence on globally toggled system services (e.g., VoiceOver) in production. | ||
| - Work reliably in background-capable scenarios and during animations (e.g., during view transitions). |
There was a problem hiding this comment.
m: I'm not 100% sure what background-capable scenarios are. Can you give an example? Do you mean for example a phone call or the app suddenly moving to the background?
| The process when taking a screenshot or session replay frame is as follows: | ||
|
|
||
| ```mermaid | ||
| flowchart TD | ||
| A[**Root View/Window**] | ||
| B[**Full-Screen Screenshot**<br/>UIImage from root view/window] | ||
| C[**Region Detection Engine**<br/>Accessibility, View Hierarchy, ML] | ||
| D[**Mask Renderer**<br/>Solid overlay of redact regions] | ||
| E[**Masked Output Image**] | ||
|
|
||
| A --> B --> C --> D --> E | ||
| ``` |
There was a problem hiding this comment.
Thanks for the diagram. Makes it easier to understand.
| > ![INFO] | ||
| > As of Nov 13th, 2025, this is the default implementation for identifying which areas of a view hierarchy should be masked during screenshot or session replay capture. |
There was a problem hiding this comment.
l: I'm unsure if that's the formatting you want to have
| > ![INFO] | ||
| > As of Nov 13th, 2025, this is the default implementation for identifying which areas of a view hierarchy should be masked during screenshot or session replay capture. | ||
|
|
||
| The `SentryUIRedactBuilder` is the current default implementation for identifying which areas of a view hierarchy should be masked during screenshot or session replay capture. |
There was a problem hiding this comment.
l: Maybe you could link to the class in the repo here.
|
|
||
| - A typical YOLO-style detection model can be 3-5MB, directly adding to the app's download size and increasing SDK install size | ||
| - Cannot improve detection accuracy without shipping a new SDK version, as model updates require SDK updates | ||
| - All users download the model regardless of whether they use session replay |
There was a problem hiding this comment.
m: We could solve this by putting SR into an extra package. Or maybe even an extra package or repository just for this. I would start with an extra repo or package. It's the easiest option IMO. We can also change it later.
|
|
||
| **Known Issues:** | ||
|
|
||
| During initial testing, generated output geometry did not work correctly when using a real device, requiring additional debugging and coordinate system validation. The implementation uses Vision framework's coordinate normalization to help mitigate these issues, but careful testing across device types and orientations is essential. |
There was a problem hiding this comment.
h: It's not 100% clear to me what generated output geometry did not work correctly exactly means. So when this doesn't work correctly, does it mean that masking is broken or that the output doesn't work at all? Or to rephrase: if this doesn't work correctly, do we potentially leak PII? Can we detect if it doesn't work?
|
|
||
| **Performance Concerns:** | ||
|
|
||
| Devices without Neural Engine (iPhone 7 and earlier) will fall back to CPU-only inference, which may be considerably slower. Continuous ML inference during session replay may impact battery life and trigger thermal throttling on older devices. These concerns require benchmarking across device generations to establish performance baselines. |
There was a problem hiding this comment.
m: One option to solve this would be simply turning it off on older device or to add an opt in.
| **Bring-Your-Own-Model (BYOM) Approach:** | ||
|
|
||
| To enable domain-specific optimization, a repository with training setup will be provided so customers can clone, edit, and build their own models. Customers can add domain-specific test data from their own apps (e.g., banking apps with credit card forms) and train specialized models optimized for their specific UI patterns and sensitive content types. The training setup can also be used to derive models compatible for Android, enabling cross-platform model sharing. |
There was a problem hiding this comment.
m: It's great that we have this here. I would maybe add a note that this could significantly complicate debugging masking issues.
|
|
||
| ### Limitations and Considerations | ||
|
|
||
| **Best-Effort Approach:** |
There was a problem hiding this comment.
Since we changed our official approach towards masking to best effort, I think that's OK.
This PR contains a document and work-in-progress files from prototyping alternative masking approaches.
We are only going to merge the document
develop-docs/VIEW_MASKING_STRATEGIES.mdand all other files are just for context.Please only review the document
#skip-changelog