Skip to content

Commit 04938d0

Browse files
committed
Add new figs and thumbnail
1 parent 84b1f1c commit 04938d0

16 files changed

+454
-26
lines changed

β€Ž_posts/2025-11-19-encryption-in-duckdb.mdβ€Ž

Lines changed: 46 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -53,11 +53,25 @@ After the main database header, DuckDB stores two 4KB database headers that cont
5353

5454
Blocks in DuckDB are by default 256KB, but their size is configurable. At the start of each *plaintext* block there is an 8-byte block header, which stores an 8-byte checksum. The checksum is a simple calculation that is often used in database systems to check for any corrupted data.
5555

56-
<img src="{% link images/blog/encryption/checksum.png %}" width="400" />
56+
<img src="{% link images/blog/encryption/plaintext-block-light.svg %}"
57+
alt="Plaintext block"
58+
class="lightmode-img"
59+
/>
60+
<img src="{% link images/blog/encryption/plaintext-block-dark.svg %}"
61+
alt="Plaintext block"
62+
class="darkmode-img"
63+
/>
5764

5865
For encrypted blocks however, its block header consists of 64-bytes instead of 8 bytes for the checksum. The block header for encrypted blocks contains a 16-byte *nonce/IV* and, optionally, a 16-byte *tag*, depending on which encryption cipher is used. The nonce and tag are stored in plaintext, but the checksum is encrypted for better security. Note that the block header always needs to be 8-bytes aligned to calculate the checksum.
5966

60-
<img src="{% link images/blog/encryption/encrypted-blocks.png %}" width="400" />
67+
<img src="{% link images/blog/encryption/encrypted-block-light.svg %}"
68+
alt="Encrypted block"
69+
class="lightmode-img"
70+
/>
71+
<img src="{% link images/blog/encryption/encrypted-block-dark.svg %}"
72+
alt="Encrypted block"
73+
class="darkmode-img"
74+
/>
6175

6276
### Write-Ahead-Log Encryption
6377

@@ -66,7 +80,7 @@ The write ahead log (WAL) in database systems is a crash recovery mechanism to e
6680
In DuckDB, you can force the creation of a WAL by setting
6781

6882
```sql
69-
PRAGMA disable_checkpoint_on_shutdown
83+
PRAGMA disable_checkpoint_on_shutdown;
7084
PRAGMA wal_autocheckpoint = '1TB';
7185
```
7286

@@ -85,11 +99,26 @@ If we now close the DuckDB process, we can see that there is a `.wal` file shown
8599

86100
Before writing new entries (inserts, updates, deletes) to the database, these entries are essentially logged and appended to the WAL. Only *after* logged entries are flushed to disk, a transaction is considered as committed. A plaintext WAL entry has the following structure:
87101

88-
<img src="{% link images/blog/encryption/plaintext-wal-entry.png %}" width="400" />
102+
<img src="{% link images/blog/encryption/plaintext-wal-entry-light.svg %}"
103+
alt="Plaintext block"
104+
class="lightmode-img"
105+
/>
106+
<img src="{% link images/blog/encryption/plaintext-wal-entry-dark.svg %}"
107+
alt="Plaintext block"
108+
class="darkmode-img"
109+
/>
110+
89111

90112
Since the WAL is append-only, we encrypt a WAL entry *per value*. For AES-GCM this means that we append a nonce and a tag to each entry. The structure in which we do this is depicted in [image]. When we serialize an encrypted entry to the encrypted WAL, we first store the length in plaintext, because we need to know how many bytes we should decrypt. The length is followed by a nonce, which on its turn is followed by the encrypted checksum and the encrypted entry itself. After the entry, a 16-byte tag is stored for verification.
91113

92-
<img src="{% link images/blog/encryption/encrypted-wal-entry.png %}" width="400" />
114+
<img src="{% link images/blog/encryption/encrypted-wal-entry-light.svg %}"
115+
alt="Plaintext block"
116+
class="lightmode-img"
117+
/>
118+
<img src="{% link images/blog/encryption/encrypted-wal-entry-dark.svg %}"
119+
alt="Plaintext block"
120+
class="darkmode-img"
121+
/>
93122

94123
Encrypting the WAL is triggered by default when an encryption key is given for any (un)encrypted database.
95124

@@ -183,13 +212,23 @@ ATTACH 'encrypted.duckdb' AS encrypted (
183212
);
184213
```
185214

186-
To keep track of which databases are encrypted, you can query this by
215+
To keep track of which databases are encrypted, you can query this by running:
187216

188217
```sql
189218
FROM duckdb_databases();
190219
```
191220

192-
Which will show which databases are encrypted, and which cipher is used.
221+
This will show which databases are encrypted, and which cipher is used:
222+
223+
| database_name | database_oid | path | … | encrypted | cipher |
224+
|---------------|--------------|------------------------|---|-----------|--------|
225+
| encrypted | 2103 | encrypted.duckdb | … | true | GCM |
226+
| unencrypted | 2050 | unencrypted.duckdb | … | false | NULL |
227+
| memory | 592 | NULL | … | false | NULL |
228+
| system | 0 | NULL | … | false | NULL |
229+
| temp | 1995 | NULL | … | false | NULL |
230+
231+
_5 rows β€” 10 columns (5 shown)_
193232

194233
## Implementation and Performance
195234

-1.91 KB
Binary file not shown.
-106 KB
Binary file not shown.

β€Žimages/blog/encryption/encrypted-block-dark.svgβ€Ž

Lines changed: 55 additions & 0 deletions
Loading

β€Žimages/blog/encryption/encrypted-block-light.svgβ€Ž

Lines changed: 55 additions & 0 deletions
Loading
-15.4 KB
Binary file not shown.

β€Žimages/blog/encryption/encrypted-wal-entry-dark.svgβ€Ž

Lines changed: 62 additions & 0 deletions
Loading

β€Žimages/blog/encryption/encrypted-wal-entry-light.svgβ€Ž

Lines changed: 62 additions & 0 deletions
Loading
-8.86 KB
Binary file not shown.
Lines changed: 24 additions & 0 deletions
Loading

0 commit comments

Comments
Β (0)