Attaching to AWS Glue fails with 403 forbidden

[din14970]

Forgive me if this is a duplicate or if I've missed something on my end.
I've looked at other issues like #517 and #454, but it seems like I'm getting a different issue.

I'm trying to connect duckdb to Iceberg tables on S3 and registered in Glue.
We are not using S3 tables, but I've seen other issues where this still seems to work, or at least others get further than me.
Following the docs and some of the modifications, I always get stuck at trying to attach the Glue catalog:

con.execute(f"""
ATTACH '<ACCOUNT>' AS glue_catalog (
    TYPE iceberg,
    ENDPOINT_TYPE 'glue'
)
""")

This raises the following error:

Request to 'glue.<region>.amazonaws.com/iceberg/v1/config?warehouse=<ACCOUNT>' returned a non-200 status code (Forbidden_403), with reason: 

The reason is just empty.

I can connect and interact with the tables with pyiceberg, and I can read parquet files on S3 with the duckdb connection, so the problem should not be the credentials.

My environment:

duckdb                        1.3.2
duckdb-extension-aws          1.3.2
duckdb-extension-httpfs       1.3.2
duckdb-extension-iceberg      1.3.2
duckdb-extension-parquet      1.3.2
duckdb-extensions             1.3.2

Could someone suggest what the issue might be? Many thanks.

[ken5scal]

@din14970 just letting you know, #517 is experimented with duckdb v1.4.0 and some core_nightly versions

D select extension_name, extension_version, install_mode, installed_from  from duckdb_extensions() where installed = true;
┌────────────────┬───────────────────┬───────────────────┬────────────────┐
│ extension_name │ extension_version │   install_mode    │ installed_from │
│    varchar     │      varchar      │      varchar      │    varchar     │
├────────────────┼───────────────────┼───────────────────┼────────────────┤
│ autocomplete   │ v1.4.0            │ STATICALLY_LINKED │                │
│ avro           │ 0c97a61           │ REPOSITORY        │ core           │
│ aws            │ 7cacaf6           │ REPOSITORY        │ core_nightly   │
│ core_functions │ v1.4.0            │ STATICALLY_LINKED │                │
│ httpfs         │ 8356a90           │ REPOSITORY        │ core_nightly   │
│ iceberg        │ 4f3c5499          │ REPOSITORY        │ core_nightly   │
│ icu            │ v1.4.0            │ STATICALLY_LINKED │                │
│ json           │ v1.4.0            │ STATICALLY_LINKED │                │
│ parquet        │ v1.4.0            │ STATICALLY_LINKED │                │
│ shell          │ v1.4.0            │ STATICALLY_LINKED │                │
├────────────────┴───────────────────┴───────────────────┴────────────────┤
│ 10 rows                                                       4 columns │

[Tmonster]

@din14970 have you enabled external engines to access data in s3 locations as described in this comment?

That is the main reason I can think of as to why you get a 403.

[din14970]

@din14970 have you enabled external engines to access data in s3 locations as described in this comment?

That is the main reason I can think of as to why you get a 403.

I have tried this but it has no effect. AFAIK, we also are not using lake formation, just Glue databases, tables and regular S3 buckets.

I have tried with admin credentials to the AWS account and now ATTACH works, so it must be something missing in the policies associated with the normal roles. Any ideas on what permissions are required for duckdb? I'm curious why it works flawlessly with pyiceberg but not duckdb.

[nicornk]

My experience is that even though you might not explicitly use LakeFormation, iceberg+glue seems to always require it.

Given that root credentials work I would recommend you try assigning LF permissions to your IAM role and see if that helps.