Skip to content

Commit 4f1af83

Browse files
LogofileLogofile
authored
Merge pull request #201 from Logofile/sync
Documentation update
2 parents bedcd7a + d225f1d commit 4f1af83

File tree

10 files changed

+308
-94
lines changed

10 files changed

+308
-94
lines changed

‎content/editions/features.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -472,7 +472,7 @@ message Msg {
472472

473473
This feature determines how generated code should treat string fields. This
474474
replaces the `ctype` option from proto2 and proto3, and offers a new
475-
`string_view` feature. In Edition 2023, either `ctype` or `string_view` may be
475+
`string_view` feature. In Edition 2023, either `ctype` or `string_type` may be
476476
specified on a field, but not both.
477477

478478
**Values available:**

‎content/includes/version-tables.css‎

Lines changed: 7 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -95,36 +95,36 @@ table.version-chart td.active {
9595
/* latest release column */
9696
/*
9797
* How to advance the selection of the latest release:
98-
* Replace class 'y24q3' in the following selectors with 'y24q4' (the class
98+
* Replace class 'y24q4' in the following selectors with 'y25q1' (the class
9999
* referring to the quarter of the next release). Please also update this
100100
* instruction as a courtesy to the next maintainer.
101101
*/
102102

103103
/* visually replace 'yyQq' heading with string 'Latest' */
104-
table.version-chart th.y24q3 span {
104+
table.version-chart th.y24q4 span {
105105
display: none;
106106
}
107-
table.version-chart th.y24q3::after {
107+
table.version-chart th.y24q4::after {
108108
content: "Latest"
109109
}
110110

111111
/* draw a focus rectangle around the latest release column */
112-
table.version-chart th.y24q3 {
112+
table.version-chart th.y24q4 {
113113
border-top: 2px solid #e06666 !important;
114114
border-left: 2px solid #e06666 !important;
115115
border-right: 2px solid #e06666 !important;
116116
}
117-
table.version-chart td.y24q3 {
117+
table.version-chart td.y24q4 {
118118
font-weight: bold;
119119
border-left: 2px solid #e06666 !important;
120120
border-right: 2px solid #e06666 !important;
121121
}
122-
table.version-chart tr:last-child td.y24q3 {
122+
table.version-chart tr:last-child td.y24q4 {
123123
border-bottom: 2px solid #e06666 !important;
124124
}
125125

126126
/* future release columns */
127-
table.version-chart td:not(:has(~ .y24q3)):not(.y24q3) {
127+
table.version-chart td:not(:has(~ .y24q4)):not(.y24q4) {
128128
font-style: italic;
129129
}
130130

‎content/news/2024-12-04.md‎

Lines changed: 77 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,77 @@
1+
+++
2+
title = "Changes announced December 4, 2024"
3+
linkTitle = "December 4, 2024"
4+
toc_hide = "true"
5+
description = "Changes announced for Protocol Buffers on December 4, 2024."
6+
type = "docs"
7+
+++
8+
9+
We are planning to modify the Protobuf debug APIs (including Protobuf
10+
AbslStringify, `proto2::ShortFormat`, `proto2::Utf8Format`,
11+
`Message::DebugString`, `Message::ShortDebugString`, `Message::Utf8DebugString`)
12+
in v30 to redact sensitive fields annotated by `debug_redact`; the outputs of
13+
these APIs will contain a per-process randomized prefix, and so will no longer
14+
be parseable by Protobuf TextFormat Parsers.
15+
16+
## Motivation
17+
18+
Currently Protobuf debug APIs print every field in a proto into human-readable
19+
formats. This may lead to privacy incidents where developers accidentally log
20+
Protobuf debug outputs containing sensitive fields.
21+
22+
## How to Annotate Sensitive Fields
23+
24+
There are two ways to mark fields sensitive:
25+
26+
* Mark a field with the field option `debug_redact = true`, directly.
27+
28+
```proto
29+
message Foo {
30+
optional string secret = 1 [debug_redact = true];
31+
}
32+
```
33+
34+
* If you have already defined a field annotation of type Enum by extending
35+
`proto2.FieldOptions`, and certain values of this annotation are used to
36+
annotate fields you would like to redact, then you can annotate these values
37+
with `debug_redact = true`. All the fields that have been annotated with
38+
such values will be redacted.
39+
40+
```proto
41+
package my.package;
42+
43+
extend proto2.FieldOptions {
44+
# The existing field annotation
45+
optional ContentType content_type = 1234567;
46+
};
47+
48+
enum ContentType {
49+
PUBLIC = 0;
50+
SECRET = 1 [debug_redact = true];
51+
};
52+
53+
message Foo {
54+
# will not be redacted
55+
optional string public_info = 1 [
56+
(my.package.content_type) = PUBLIC
57+
];
58+
# will be redacted
59+
optional string secret = 1 [
60+
(my.package.content_type) = SECRET
61+
];
62+
}
63+
```
64+
65+
## New Debug Format
66+
67+
Compared to the existing debug format, the new debug format has two major
68+
differences:
69+
70+
* The sensitive fields annotated with `debug_redact` are redacted
71+
automatically in the output formats
72+
* The output formats will contain a per-process randomized prefix, which will
73+
make them no longer be parsable by TextFormat parsers.
74+
75+
Note that the second change is true regardless of whether the proto contains
76+
sensitive fields or not, which ensures that any debug output always cannot be
77+
deserialized regardless of the proto content.

‎content/news/_index.md‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -20,6 +20,8 @@ New news topics will also be published to the
2020
The following news topics provide information in the reverse order in which it
2121
was released.
2222

23+
* [December 4, 2024](/news/2024-12-04) - `DebugString`
24+
replaced
2325
* [November 7, 2024](/news/2024-11-07) - More breaking
2426
changes in the upcoming 30.x release
2527
* [October 2, 2024](/news/2024-10-02) - Breaking

‎content/news/v30.md‎

Lines changed: 12 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -43,6 +43,18 @@ dependencies.
4343
If you use installed packages, you won't be affected. It could break some CMake
4444
workflows.
4545

46+
### Modify Debug APIs to Redact Sensitive Fields {#debug-redaction}
47+
48+
We are planning to modify the Protobuf debug APIs (including Protobuf
49+
AbslStringify, `proto2::ShortFormat`, `proto2::Utf8Format`,
50+
`Message::DebugString`, `Message::ShortDebugString`, `Message::Utf8DebugString`)
51+
in v30 to redact sensitive fields annotated by `debug_redact`; the outputs of
52+
these APIs will contain a per-process randomized prefix, and so will no longer
53+
be parseable by Protobuf TextFormat Parsers.
54+
55+
Read more about this in the
56+
[news article released November 21, 2024](/news/2024-11-21).
57+
4658
### Remove Deprecated APIs {#remove-deprecated}
4759

4860
v30 will remove the following public runtime APIs, which have been marked

‎content/programming-guides/deserialize-debug.md‎

Lines changed: 120 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,120 @@
1+
+++
2+
title = "Deserializing Debug Proto Representations"
3+
weight = 89
4+
description = "How to log debugging information in Protocol Buffers."
5+
type = "docs"
6+
+++
7+
8+
From version 29.x, `DebugString` APIs (`proto2::DebugString`,
9+
`proto2::ShortDebugString`, `proto2::Utf8DebugString`) are deprecated.
10+
DebugString users should migrate to some Abseil string functions (such as
11+
`absl::StrCat`, `absl::StrFormat`, `absl::StrAppend`, AND `absl::Substitute`),
12+
Abseil logging API, and some Protobuf APIs (`proto2::ShortFormat`,
13+
`proto2::Utf8Format`) to automatically convert proto arguments into a new
14+
debugging format .
15+
16+
Unlike the Protobuf DebugString output format, the new debugging format
17+
automatically redacts sensitive fields by replacing their values with the string
18+
"[REDACTED]" (without the quotation marks). In
19+
addition, to ensure that this new output format cannot be deserialized by
20+
Protobuf TextFormat parsers, regardless of whether the underlying proto contains
21+
SPII fields, we add a set of randomized links pointing to this article
22+
and a randomized-length whitespace sequence. The new debugging format looks as
23+
follows:
24+
25+
```none
26+
go/nodeserialize
27+
spii_field: [REDACTED]
28+
normal_field: "value"
29+
```
30+
31+
Note that the new debugging format is only different from the output format of
32+
DebugString format in two ways:
33+
34+
* The URL prefix
35+
* The values of SPII fields are replaced by
36+
"[REDACTED]" (without the quotes)
37+
38+
The new debugging format never removes any field names; it only replaces the
39+
value with
40+
"[REDACTED]" if the field is considered sensitive.
41+
**If you don't see certain fields in the output, it is because those fields are
42+
not set in the proto.**
43+
44+
**Tip:** If you see only the URL and nothing else, your proto is empty!
45+
46+
## Why is this URL here?
47+
48+
We want to make sure nobody deserializes human-readable representations of a
49+
protobuf message intended for humans debugging a system. Historically,
50+
`.DebugString()` and `TextFormat` were interchangeable, and existing systems use
51+
DebugString to transport and store data.
52+
53+
We want to make sure sensitive data does not accidentally end up in logs.
54+
Therefore, we are transparently redacting some field values from protobuf
55+
messages before turning them into a string
56+
("[REDACTED]"). This reduces the security & privacy
57+
risk of accidental logging, but risks data loss if other systems deserialize
58+
your message. To address this risk, we are intentionally splitting the
59+
machine-readable TextFormat from the human-readable debug format to be used in
60+
log messages.
61+
62+
### Why are there links in my web page? Why is my code producing this new "debug representation"?
63+
64+
This is intentional, to make the "debug representation" of your protos
65+
(produced, for example, by logging) incompatible with TextFormat. We want to
66+
prevent anyone from depending on debugging mechanisms to transport data between
67+
programs. Historically, the debug format (generated by the DebugString APIs) and
68+
TextFormat have been incorrectly used in a interchangeable fashion. We hope this
69+
intentional effort will prevent that going forward.
70+
71+
We intentionally picked a link over less visible format changes to get an
72+
opportunity to provide context. This might stand out in UIs, such as if you
73+
display status information on a table in a webpage. You may use
74+
`TextFormat::PrintToString` instead, which will not redact any information and
75+
preserves formatting. However, use this API cautiously -- there are no built in
76+
protections. As a rule of thumb, if you are writing data to debug logs, or
77+
producing status messages, you should continue to use the Debug Format with the
78+
link. Even if you are currently not handling sensitive data, keep in mind that
79+
systems can change and code gets re-used.
80+
81+
### I tried converting this message into TextFormat, but I noticed the format changes every time my process restarts.
82+
83+
This is intentional. Don't attempt to parse the output of this debug format. We
84+
reserve the right to change the syntax without notice. The debug format syntax
85+
randomly changes per process to prevent inadvertent dependencies. If a syntactic
86+
change in the debug format would break your system, chances are you shouldn't
87+
use the debug representation of a proto.
88+
89+
## FAQ
90+
91+
### Can I Just Use TextFormat Everywhere?
92+
93+
Don't use TextFormat for producing log messages. This will bypass all built-in
94+
protections, and you risk accidentally logging sensitive information. Even if
95+
your systems are currently not handling any sensitive data, this can change in
96+
the future.
97+
98+
Distinguish logs from information that's meant for further processing by other
99+
systems by using either the debug representation or TextFormat as appropriate.
100+
101+
### I Want to Write Configuration Files That Need to Be Both Human-Readable And Machine-Readable
102+
103+
For this use case, you can use TextFormat explicitly. You are responsible for
104+
making sure your configuration files don't contain any PII.
105+
106+
### I Am Writing a Unit Test, and Want to Compare Debugstring in a Test Assertion
107+
108+
If you want to compare protobuf values, use `MessageDifferencer` like in the
109+
following:
110+
111+
```cpp
112+
using google::protobuf::util::MessageDifferencer;
113+
...
114+
MessageDifferencer diff;
115+
...
116+
diff.Compare(foo, bar);
117+
```
118+
119+
Besides ignoring formatting and field order differences, you will also get
120+
better error messages.

‎content/programming-guides/dos-donts.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -183,7 +183,7 @@ from repeated to scalar will result in the last deserialized value "winning."
183183

184184
Going from scalar to repeated is OK in proto2 and in proto3 with
185185
`[packed=false]` because for binary serialization the scalar value becomes a
186-
one-element list .
186+
one-element list.
187187

188188
<a id="do-follow-the-style-guide-for-generated-code"></a>
189189

0 commit comments

Comments
 (0)