Custom S3 endpoint is not used with vended credentials during write operations

[coca-alex]

Hi!
I have MinIO as S3-compatible storage and custom REST API catalog. I want to use vended credentials during both read and write operations.

DuckDB version is 1.4.2

install iceberg;
load iceberg;

select extension_name, extension_version, install_mode
from duckdb_extensions()
where installed = true;

┌────────────────┬───────────────────┬───────────────────┐
│ extension_name │ extension_version │   install_mode    │
│    varchar     │      varchar      │      varchar      │
├────────────────┼───────────────────┼───────────────────┤
│ autocomplete   │ v1.4.2            │ STATICALLY_LINKED │
│ avro           │ 93da8a1           │ REPOSITORY        │
│ core_functions │ v1.4.2            │ STATICALLY_LINKED │
│ httpfs         │ 041a782           │ REPOSITORY        │
│ iceberg        │ 5f5cd041          │ REPOSITORY        │
│ icu            │ v1.4.2            │ STATICALLY_LINKED │
│ json           │ v1.4.2            │ STATICALLY_LINKED │
│ parquet        │ v1.4.2            │ STATICALLY_LINKED │
│ shell          │ v1.4.2            │ STATICALLY_LINKED │
└────────────────┴───────────────────┴───────────────────┘

-- I tried to use S3 secret manager:
create or replace secret secret (
  type s3,
  endpoint 'localhost:9000',
  url_style 'path'
  );

-- Next, create Iceberg secret:
create secret iceberg_secret (
  type iceberg,
  token '********'
);

-- Attach Iceberg catalog:
attach 'warehouse' as iceberg_catalog (
  type iceberg,
  secret iceberg_secret,
  endpoint 'https://localhost:8080/iceberg/****',
  default_region 'eu-central-1');

-- Try to read:
select id from iceberg_catalog.test.test_table;

-- Got an error:
IO Error:
Could not establish connection error for HTTP GET to 'https://iceberg-data.s3.eu-central-1.amazonaws.com/warehouse/test/test_table-8a0779bda7ca40a0853885d754551900/metadata/snap-3227326154498549852-1-e4da9111-a9a7-45e1-8df4-53abc7bf9239.avro'

-- Try to write:
insert into iceberg_catalog.test.test_table values (1);

-- Got an error:
IO Error:
Could not establish connection error for HTTP PUT to 'https://s3.eu-central-1.amazonaws.com/iceberg-data/warehouse/test/test_table/data/019a984a-102a-72d2-9aa7-1ca391be66cf.parquet'

-- Then, I tried to set legacy S3 environment variables:
set s3_region='eu-central-1';
set s3_url_style='path';
set s3_endpoint='localhost:9000';

-- Try to read:
select id from iceberg_catalog.test.test_table;

-- Good:
┌─────────┐
│   id    │
│ varchar │
├─────────┤
│ 0 rows  │
└─────────┘

-- Try to write:
insert into iceberg_catalog.test.test_table values (1);

-- Got an error:
TransactionContext Error:
Failed to commit: Failed to commit Iceberg transaction: Could not establish connection error for HTTP HEAD to 'https://iceberg-data.s3.eu-central-1.amazonaws.com/warehouse/test/test_table/data/e9522eb6-00d0-467f-ac9b-066277dd454e-m0.avro'

Reading with vended credentials works only with legacy S3 environment variables for me.
I suppose this is because for Glue and S3 tables, S3 endpoint is set in this and this parts of the code, but I couldn’t find where it’s set for others.

Writing with vended credentials always fails.

Also, I've tried previous versions of DuckDB — 1.4.1 and 1.4.0 — but the behavior was the same.

[Tmonster]

Hi,

Thanks for filing the issue. I don't think this is a bug, if you have vended credentials, a custom s3 endpoint should not overwrite the credentials returned via vended credentials.

If you are using your own custom Iceberg Rest Catalog at https://localhost:8080/iceberg/**** that supports vended credentials, DuckDB will try to create a secret with the vended credentials to access storage. That is most likely why the endpoints you see are https://iceberg-data.s3.eu-central-1.amazonaws.com/***.avro and not localhost:9000/**.avro.

If you want the s3 secret secret to be used to access your storage, I would suggest passing ACCESS_DELEGATION_MODE 'none', on attach. Then DuckDB-Iceberg will look at your custom s3 secret to access storage.

Also, interesting that the URLs for HEAD and PUT have different prefixes,
GET

https://s3.eu-central-1.amazonaws.com/***

vs.
PUT

https://iceberg-data.s3.eu-central-1.amazonaws.com/

Maybe something is wrong with the vended credential prefix, or DuckDB is parsing the prefix incorrectly.

If you are still experiencing issues, it would be helpful if you could share your catalog implementation, that makes reproduction/debugging easier.