Skip to content

Conversation

@badalprasadsingh
Copy link
Collaborator

@badalprasadsingh badalprasadsingh commented Oct 15, 2025

Description

Even when, ADLS Endpoint was being passed, OLake was not able to handle ADLSFileio.

The unecessary if condition is removed as org.apache.iceberg.io.ResolvingFileIO will be able to handle the FileIO for respective GCS, AWS or Azure Cloud providers.

Fixes #589

How to test it?

Follow the doc: https://olake.io/docs/writers/iceberg/azure to run your sync using OLake.

Then, query the table using Spark:

version: "3.8"

services:
  spark:
    image: tabulario/spark-iceberg:latest
    container_name: spark-iceberg
    ports:
      - "4040:4040"  
      - "8888:8888"  
    volumes:
      - ./workspace:/home/iceberg/workspace
    environment:
      - SPARK_DRIVER_MEMORY=2g
      - SPARK_EXECUTOR_MEMORY=2g
    tty: true
docker compose up -d
docker exec -it spark-iceberg bash
spark-sql \
  --packages org.apache.iceberg:iceberg-spark-3.5_2.12:1.10.0 \
  --conf spark.sql.extensions=org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions \
  --conf spark.sql.catalog.my_azure_db=org.apache.iceberg.spark.SparkCatalog \
  --conf spark.sql.catalog.my_azure_db.type=rest \
  --conf spark.sql.catalog.my_azure_db.uri=http://<LAKEKEEPER_ENDPOINT>:8181/catalog \
  --conf spark.sql.catalog.my_azure_db.warehouse=olake_warehouse \
  --conf spark.sql.catalog.my_azure_db.vended-credentials-enabled=true \
  --conf spark.sql.defaultCatalog=my_azure_db

Replace LAKEKEEPER_ENDPOINT with your correct REST_ENDPOINT

SHOW DATABASES;
SELECT * FROM <schema>.<table>;

Type of change

  • Bug fix (non-breaking change which fixes an issue)
  • New feature (non-breaking change which adds functionality)
  • Breaking change (fix or feature that would cause existing functionality to not work as expected)
  • This change requires a documentation update

How Has This Been Tested?

  • Lakekeeper (REST) + Azure ADLS
  • Glue + S3
  • Iceberg-fixture (REST) + Minio

Screenshots or Recordings

image

Documentation

  • Documentation Link: [link to README, olake.io/docs, or olake-docs]
  • N/A (bug fix, refactor, or test changes only)

Signed-off-by: badalprasadsingh <[email protected]>
Signed-off-by: badalprasadsingh <[email protected]>
@badalprasadsingh badalprasadsingh changed the title Fix: Azure ADLS Lakekeeper Rest Catalog Issue fix: Azure ADLS Lakekeeper Rest Catalog Issue Oct 16, 2025
Signed-off-by: badalprasadsingh <[email protected]>
Signed-off-by: badalprasadsingh <[email protected]>
Signed-off-by: badalprasadsingh <[email protected]>
Signed-off-by: badalprasadsingh <[email protected]>
Signed-off-by: badalprasadsingh <[email protected]>
Signed-off-by: badalprasadsingh <[email protected]>
@shubham19may shubham19may merged commit 9c23984 into staging Oct 17, 2025
9 of 11 checks passed
Utkarsh9571 pushed a commit to Utkarsh9571/olake that referenced this pull request Nov 1, 2025
badalprasadsingh added a commit that referenced this pull request Nov 10, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants