You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: _posts/2025-11-19-encryption-in-duckdb.md
+46-7Lines changed: 46 additions & 7 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -53,11 +53,25 @@ After the main database header, DuckDB stores two 4KB database headers that cont
53
53
54
54
Blocks in DuckDB are by default 256KB, but their size is configurable. At the start of each *plaintext* block there is an 8-byte block header, which stores an 8-byte checksum. The checksum is a simple calculation that is often used in database systems to check for any corrupted data.
55
55
56
-
<imgsrc="{% link images/blog/encryption/checksum.png %}"width="400" />
56
+
<img src="{% link images/blog/encryption/plaintext-block-light.svg %}"
57
+
alt="Plaintext block"
58
+
class="lightmode-img"
59
+
/>
60
+
<img src="{% link images/blog/encryption/plaintext-block-dark.svg %}"
61
+
alt="Plaintext block"
62
+
class="darkmode-img"
63
+
/>
57
64
58
65
For encrypted blocks however, its block header consists of 64-bytes instead of 8 bytes for the checksum. The block header for encrypted blocks contains a 16-byte *nonce/IV* and, optionally, a 16-byte *tag*, depending on which encryption cipher is used. The nonce and tag are stored in plaintext, but the checksum is encrypted for better security. Note that the block header always needs to be 8-bytes aligned to calculate the checksum.
59
66
60
-
<imgsrc="{% link images/blog/encryption/encrypted-blocks.png %}"width="400" />
67
+
<img src="{% link images/blog/encryption/encrypted-block-light.svg %}"
68
+
alt="Encrypted block"
69
+
class="lightmode-img"
70
+
/>
71
+
<img src="{% link images/blog/encryption/encrypted-block-dark.svg %}"
72
+
alt="Encrypted block"
73
+
class="darkmode-img"
74
+
/>
61
75
62
76
### Write-Ahead-Log Encryption
63
77
@@ -66,7 +80,7 @@ The write ahead log (WAL) in database systems is a crash recovery mechanism to e
66
80
In DuckDB, you can force the creation of a WAL by setting
67
81
68
82
```sql
69
-
PRAGMA disable_checkpoint_on_shutdown
83
+
PRAGMA disable_checkpoint_on_shutdown;
70
84
PRAGMA wal_autocheckpoint ='1TB';
71
85
```
72
86
@@ -85,11 +99,26 @@ If we now close the DuckDB process, we can see that there is a `.wal` file shown
85
99
86
100
Before writing new entries (inserts, updates, deletes) to the database, these entries are essentially logged and appended to the WAL. Only *after* logged entries are flushed to disk, a transaction is considered as committed. A plaintext WAL entry has the following structure:
87
101
88
-
<imgsrc="{% link images/blog/encryption/plaintext-wal-entry.png %}"width="400" />
102
+
<img src="{% link images/blog/encryption/plaintext-wal-entry-light.svg %}"
103
+
alt="Plaintext block"
104
+
class="lightmode-img"
105
+
/>
106
+
<img src="{% link images/blog/encryption/plaintext-wal-entry-dark.svg %}"
107
+
alt="Plaintext block"
108
+
class="darkmode-img"
109
+
/>
110
+
89
111
90
112
Since the WAL is append-only, we encrypt a WAL entry *per value*. For AES-GCM this means that we append a nonce and a tag to each entry. The structure in which we do this is depicted in [image]. When we serialize an encrypted entry to the encrypted WAL, we first store the length in plaintext, because we need to know how many bytes we should decrypt. The length is followed by a nonce, which on its turn is followed by the encrypted checksum and the encrypted entry itself. After the entry, a 16-byte tag is stored for verification.
91
113
92
-
<imgsrc="{% link images/blog/encryption/encrypted-wal-entry.png %}"width="400" />
114
+
<img src="{% link images/blog/encryption/encrypted-wal-entry-light.svg %}"
115
+
alt="Plaintext block"
116
+
class="lightmode-img"
117
+
/>
118
+
<img src="{% link images/blog/encryption/encrypted-wal-entry-dark.svg %}"
119
+
alt="Plaintext block"
120
+
class="darkmode-img"
121
+
/>
93
122
94
123
Encrypting the WAL is triggered by default when an encryption key is given for any (un)encrypted database.
95
124
@@ -183,13 +212,23 @@ ATTACH 'encrypted.duckdb' AS encrypted (
183
212
);
184
213
```
185
214
186
-
To keep track of which databases are encrypted, you can query this by
215
+
To keep track of which databases are encrypted, you can query this by running:
187
216
188
217
```sql
189
218
FROM duckdb_databases();
190
219
```
191
220
192
-
Which will show which databases are encrypted, and which cipher is used.
221
+
This will show which databases are encrypted, and which cipher is used:
0 commit comments