Revamp MotherDuck settings and secret management (#668)

# Revamp MotherDuck settings and secret management

This PR removes any GUC setting for MotherDuck and replaces it with a
combination of `SERVER` and `USER MAPPING` for a newly implemented
`pg_duckdb` Foreign Data Wrapper.

## New MotherDuck setup:

Users may now define one (and at most one) `SERVER` for `motherduck`:
```sql
CREATE SERVER <server name> 
FOREIGN DATA WRAPPER pg_duckdb
TYPE 'motherduck'
[OPTIONS (
     [tables_owner_role 'some user'],	-- the role bgw assigns to MD tables 
										-- (by default server creator/owner) 
     [default_database]					-- Which MD DB is synced (default "my_db")
)];
```

Once the `SERVER` is created, for MotherDuck to be enabled, it needs at
least one `USER MAPPING`:
```sql
CREATE USER MAPPING FOR <PG user> 
SERVER <FDW server name> 
OPTIONS (
  token '<some token>'
);
```

## Removed GUC settings:

The following GUC settings were thus removed and are now obsolete:

- `motherduck_enabled`: now implied by a `SERVER` and `USER MAPPING`
- `motherduck_token`: now defined in a `USER MAPPING`
- `motherduck_postgres_database`: now implied by the database in which
the `SERVER` & `USER MAPPING` are defined
- `motherduck_default_database`: now defined as an option of the
`motherduck` server

## Convenience function:

In order to enable MotherDuck support easily, we also provide the
following helper:

```sql
-- Token will be read from environment if not provided
-- Default database is `my_db` if not provided
SELECT duckdb.enable_motherduck('<token>', '<default_motherduck_db>')
```

---------

Co-authored-by: Jelte Fennema-Nio <[email protected]>

Author: Y--

Dockerfile

@@ -63,7 +63,7 @@ RUN apt-get update -qq && \
 
 # Automatically enable pg_duckdb
 RUN echo "shared_preload_libraries='pg_duckdb'" >> /usr/share/postgresql/postgresql.conf.sample
-RUN echo "CREATE EXTENSION IF NOT EXISTS pg_duckdb;" >> /docker-entrypoint-initdb.d/0001-install-pg_duckdb.sql
+COPY --chown=postgres:postgres docker/init.d/ /docker-entrypoint-initdb.d/
 
 COPY --from=builder /out /
 USER postgres

README.md

@@ -75,7 +75,7 @@ docker run -d -e POSTGRES_PASSWORD=duckdb pgduckdb/pgduckdb:16-main
 And with MotherDuck, you only need a [a MotherDuck access token][md-access-token] and then it is as simple as:
 ```shell
 $ export MOTHERDUCK_TOKEN=<your personal MD token>
-$ docker run -d -e POSTGRES_PASSWORD=duckdb -e MOTHERDUCK_TOKEN pgduckdb/pgduckdb:16-main -c duckdb.motherduck_enabled=true
+$ docker run -d -e POSTGRES_PASSWORD=duckdb -e MOTHERDUCK_TOKEN pgduckdb/pgduckdb:16-main
 ```
 
 Or you can use the docker compose in this repo:
@@ -181,37 +181,26 @@ Note: writes to Azure are not yet supported, please see [the current discussion]
 
 ### Connect with MotherDuck
 
-pg_duckdb also integrates with [MotherDuck][md]. To enable this support you first need to [generate an access token][md-access-token] and then add the following line to your `postgresql.conf` file:
+`pg_duckdb` also integrates with [MotherDuck][md].
+To enable this support you first need to [generate an access token][md-access-token].
+Then you can enable it by simply using the `enable_motherduck` convenience method:
 
-```ini
-duckdb.motherduck_token = 'your_access_token'
-```
-
-NOTE: If you don't want to store the token in your `postgresql.conf`file can also store the token in the `motherduck_token` environment variable and then explicitly enable MotherDuck support in your `postgresql.conf` file:
-
-```ini
-duckdb.motherduck_enabled = true
-```
-
-If you installed `pg_duckdb` in a different Postgres database than the default one named `postgres`, then you also need to add the following line to your `postgresql.conf` file:
-
-```ini
-duckdb.motherduck_postgres_database = 'your_database_name'
+```sql
+-- If not provided, the token will be read from the `motherduck_token` environment variable
+-- If not provided, the default MD database name is `my_db`
+SELECT duckdb.enable_motherduck('<optional token>', '<optional MD database name>');
 ```
 
-The default MotherDuck database will be easiest to use (see below for details), by default this is `my_db`. If you want to specify which MotherDuck database is your default database, then you can also add the following line to your `postgresql.conf` file:
-
-```ini
-duckdb.motherduck_default_database = 'your_motherduck_database_name'
-```
+Read more [here][md-docs] about MotherDuck integration.
 
-After doing this (and possibly restarting Postgres). You can then you create tables in the MotherDuck database by using the `duckdb` [Table Access Method][tam] like this:
+You can now create tables in the MotherDuck database by using the `duckdb` [Table Access Method][tam] like this:
 ```sql
 CREATE TABLE orders(id bigint, item text, price NUMERIC(10, 2)) USING duckdb;
 CREATE TABLE users_md_copy USING duckdb AS SELECT * FROM users;
 ```
 
 [tam]: https://www.postgresql.org/docs/current/tableam.html
+[md-docs]: docs/motherduck.md
 
 
 Any tables that you already had in MotherDuck are automatically available in Postgres. Since DuckDB and MotherDuck allow accessing multiple databases from a single connection and Postgres does not, we map database+schema in DuckDB to a schema name in Postgres.

docker/README.md

@@ -35,7 +35,7 @@ docker run -d -e POSTGRES_PASSWORD=duckdb pgduckdb/pgduckdb:16-main
 And with MotherDuck, it is as simple as:
 ```shell
 $ export MOTHERDUCK_TOKEN=<your personal MD token>
-$ docker run -d -e POSTGRES_PASSWORD=duckdb -e MOTHERDUCK_TOKEN pgduckdb/pgduckdb:16-main -c duckdb.motherduck_enabled=true
+$ docker run -d -e POSTGRES_PASSWORD=duckdb -e MOTHERDUCK_TOKEN pgduckdb/pgduckdb:16-main
 ```
 
 You can also use docker compose from duckdb/pg_duckdb:

docker/init.d/0001-install-pg_duckdb.sql

@@ -0,0 +1 @@
+CREATE EXTENSION IF NOT EXISTS pg_duckdb;

docker/init.d/0002-enable-md-pg_duckdb.sql

@@ -0,0 +1,11 @@
+-- This script is used to enable MotherDuck support if
+-- the token is provided in the environment variables.
+
+\getenv lc_token motherduck_token
+\getenv uc_token MOTHERDUCK_TOKEN
+
+\if :{?lc_token}
+    SELECT duckdb.enable_motherduck(:'lc_token'::TEXT);
+\elif :{?uc_token}
+    SELECT duckdb.enable_motherduck(:'uc_token'::TEXT);
+\endif

docs/motherduck.md

@@ -2,23 +2,19 @@
 
 ## Connect with MotherDuck
 
-pg_duckdb also integrates with [MotherDuck][md]. To enable this support you first need to [generate an access token][md-access-token] and then add the following line to your `postgresql.conf` file:
+`pg_duckdb` integrates with [MotherDuck][md] natively.
+To enable this support you first need to [generate an access token][md-access-token].
+Then you can enable it by simply using the `enable_motherduck` convenience method:
 
-```ini
-duckdb.motherduck_token = 'your_access_token'
-```
-
-NOTE: If you don't want to store the token in your `postgresql.conf`file can also store the token in the `motherduck_token` environment variable and then explicitly enable MotherDuck support in your `postgresql.conf` file:
-
-```ini
-duckdb.motherduck_enabled = true
+```sql
+-- If not provided, the token will be read from the `motherduck_token` environment variable
+-- If not provided, the default MD database name is `my_db`
+SELECT duckdb.enable_motherduck('<optional token>', '<optional MD database name>');
 ```
 
-If you installed `pg_duckdb` in a different Postgres database than the default one named `postgres`, then you also need to add the following line to your `postgresql.conf` file:
+This convenience method creates a `motherduck` `SERVER` using the `pg_duckdb` Foreign Data Wrapper, which hosts the options for this integration. It also provides an `USER MAPPING` for the current user, which stores the provided MotherDuck token (if any).
 
-```ini
-duckdb.motherduck_postgres_database = 'your_database_name'
-```
+You can refer to the [section](advanced-motherduck-configuration) below for more on the `SERVER` and `USER MAPPING` configuration.
 
 ### Non-supersuer configuration
 
@@ -55,10 +51,25 @@ CREATE TABLE users_md_copy USING duckdb AS SELECT * FROM users;
 
 Any tables that you already had in MotherDuck are automatically available in Postgres. Since DuckDB and MotherDuck allow accessing multiple databases from a single connection and Postgres does not, we map database+schema in DuckDB to a schema name in Postgres.
 
-The default MotherDuck database will be easiest to use (see below for details), by default this is `my_db`. If you want to specify which MotherDuck database is your default database, then you can also add the following line to your `postgresql.conf` file:
+The default MotherDuck database will be easiest to use (see below for details), by default this is `my_db`.
 
-```ini
-duckdb.motherduck_default_database = 'your_motherduck_database_name'
+## Advanced MotherDuck configuration
+
+If you want to specify which MotherDuck database is your default database, then you need to configure MotherDuck using a `SERVER` and a `USER MAPPING` as such:
+
+```sql
+CREATE SERVER motherduck
+TYPE 'motherduck'
+FOREIGN DATA WRAPPER duckdb
+OPTIONS (default_database '<your database>');
+
+-- You may use `::FROM_ENV::` to have the token be read from the environment variable
+CREATE USER MAPPING FOR CURRENT_USER SERVER motherduck OPTIONS (token '<your token>')
+```
+
+Note: with the `duckdb.enable_motherduck` convenience method above, you can simply do:
+```sql
+SELECT duckdb.enable_motherduck('<token>', '<default database>');
 ```
 
 The mapping of database+schema to schema name is then done in the following way:

docs/settings.md

@@ -10,45 +10,6 @@ Default: `false`
 
 Access: General
 
-## MotherDuck
-
-MotherDuck support is optional, and can be enabled an configured using these settings. Check out our [MotherDuck documentation](motherduck.md) for more information.
-
-### `duckdb.motherduck_enabled`
-
-If MotherDuck support should be enabled. `auto` means enabled if the `duckdb.motherduck_token` is set.
-
-Default: `MotherDuckEnabled::MOTHERDUCK_AUTO`
-
-Access: Needs to be in the `postgresql.conf` file and requires a restart
-
-### `duckdb.motherduck_token`
-
-The token to use for MotherDuck
-
-Default: `""`
-
-Access: Needs to be in the `postgresql.conf` file and requires a restart
-
-### `duckdb.motherduck_postgres_database`
-
-Which database to enable MotherDuck support in
-
-Default: `"postgres"`
-
-Access: Needs to be in the `postgresql.conf` file and requires a restart
-
-### `duckdb.motherduck_default_database`
-
-Which MotherDuck database to use as the default database, i.e. the one that
-gets merged with Postgres schemas instead of getting dedicated `ddb$` prefixed
-schemas. The empty string means that pg_duckdb should use the default database
-set by MotherDuck, which is currently always `my_db`.
-
-Default: `""`
-
-Access: Needs to be in the `postgresql.conf` file and requires a restart
-
 ## Security
 
 ### `duckdb.postgres_role`

include/pgduckdb/pg/string_utils.hpp

@@ -0,0 +1,13 @@
+#pragma once
+
+#include <string.h>
+
+inline bool
+AreStringEqual(const char *str1, const char *str2) {
+	return strcmp(str1, str2) == 0;
+}
+
+inline bool
+IsEmptyString(const char *str) {
+	return AreStringEqual(str, "");
+}

include/pgduckdb/pgduckdb_fdw.hpp

@@ -0,0 +1,53 @@
+#pragma once
+
+#include "pgduckdb/pg/declarations.hpp"
+
+extern "C" {
+namespace pgduckdb {
+
+Oid FindMotherDuckForeignServerOid();
+
+/*
+
+Returns the Oid of the USER MAPPING for the given user and server.
+Returns InvalidOid if no such mapping exists.
+
+Note: we cannot use PG's `GetUserMapping` because:
+- it checks the global scope if not found
+- it throws an error if no mapping is not found
+
+*/
+Oid FindUserMappingForUser(Oid user_oid, Oid server_oid);
+
+/*
+
+Returns the Oid of the `tables_owner_role` option defined in the `motherduck` SERVER
+if it exists, returns the SERVER's owner otherwise.
+
+*/
+Oid GetMotherDuckPostgresRoleOid(Oid server_oid);
+
+/*
+
+Return the `default_database` setting defined in the `motherduck` SERVER
+if it exists, returns nullptr otherwise.
+
+*/
+const char *FindMotherDuckDefaultDatabase();
+
+/*
+
+- if `pgduckdb::is_background_worker` then:
+  > returns `token` option defined in the USER MAPPING of the owner of the SERVER if it exists
+  > returns nullptr otherwise
+- if not `pgduckdb::is_background_worker` then:
+  > returns `token` option defined in the USER MAPPING of the current user if it exists
+  > returns nullptr otherwise
+
+*/
+const char *FindMotherDuckToken();
+
+const char *FindMotherDuckBackgroundCatalogRefreshInactivityTimeout();
+
+} // namespace pgduckdb
+}

include/pgduckdb/pgduckdb_guc.h

@@ -12,10 +12,5 @@ extern bool duckdb_allow_unsigned_extensions;
 extern bool duckdb_autoinstall_known_extensions;
 extern bool duckdb_autoload_known_extensions;
 extern int duckdb_max_workers_per_postgres_scan;
-extern char *duckdb_motherduck_postgres_database;
-extern int duckdb_motherduck_enabled;
-extern char *duckdb_motherduck_token;
 extern char *duckdb_postgres_role;
-extern char *duckdb_motherduck_default_database;
 extern char *duckdb_motherduck_session_hint;
-extern char *duckdb_motherduck_background_catalog_refresh_inactivity_timeout;