Skip to content

Support scoped AWS configurations #729

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 9 commits into from
Oct 6, 2025
Merged

Support scoped AWS configurations #729

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
9 commits
Select commit Hold shift + click to select a range
5e80ee7
Support scoped set AWS configurations
omus Jun 6, 2025
571dd60
Use `current_aws_config()` as default
omus Jun 9, 2025
65563da
Merge branch 'master' into cv/scoped-config2
omus Jul 3, 2025
b6e178f
Merge branch 'master' into cv/scoped-config2
omus Jul 4, 2025
05a3862
Merge branch 'master' into cv/scoped-config2
omus Jul 10, 2025
eb09166
Define `default_aws_config` for outside of scope configuration
omus Jul 17, 2025
714618d
Set project version to 1.96.0
omus Jul 17, 2025
4165596
fixup! Use `current_aws_config()` as default
omus Jul 17, 2025
d9629b3
Code review typo fixes
omus Oct 6, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
The table of contents is too big for display.
Diff view
Diff view
  •  
  •  
  •  
6 changes: 4 additions & 2 deletions Project.toml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
name = "AWS"
uuid = "fbe9abb3-538b-5e4e-ba9e-bc94f4f92ebc"
license = "MIT"
version = "1.95.0"
version = "1.96.0"

[deps]
Base64 = "2a0f44e3-6c83-55bd-87e4-b1978d98bd5f"
Expand All @@ -17,6 +17,7 @@ Mocking = "78c3b35d-d492-501b-9361-3d52fe80e533"
OrderedCollections = "bac558e1-5e72-5ebc-8fee-abe8a469f55d"
Random = "9a3f8284-a2c9-5f02-9a11-845980a1fd5c"
SHA = "ea8e919c-243c-51af-8825-aaa63cd721ce"
ScopedValues = "7e506255-f358-4e82-b7e4-beb19740aa63"
Sockets = "6462fe0b-24de-5631-8697-dd941f90decc"
URIs = "5c2747f8-b7ea-4ff2-ba2e-563bfd36b1d4"
UUIDs = "cf7118a7-6976-5b1a-9a39-7adc72f591a4"
Expand All @@ -32,10 +33,11 @@ JSON = "0.18, 0.19, 0.20, 0.21"
MbedTLS = "0.6, 0.7, 1"
Mocking = "0.7, 0.8"
OrderedCollections = "1.3"
ScopedValues = "1.3"
StableRNGs = "1"
URIs = "1"
XMLDict = "0.3, 0.4"
julia = "1.6"
julia = "1.8"

[extras]
CodecBase = "6c391c72-fb7b-5838-ba82-7cfb1bcfecbf"
Expand Down
17 changes: 13 additions & 4 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -126,13 +126,16 @@ S3.DeleteObjects(bucket_name, body) # Delete multiple objects
There are most likely other similar functions which require more intricate details in how the requests are performed, both in the S3 definitions and in other services.

## Modifying Functionality

There are sometimes situations, in which default behavior of AWS.jl might be overridden, for example when this package is used to access S3-compatible object storage of a different cloud service provider, which might have different ways of joining the endpoint url, encoding the region in the signature etc.

In many cases this can be achieved by creating a user-defined subtype of `AbstractAWSConfig` where some of the default methods are overwritten.
For example, if you want to use the S3 high-level interface to access public data from GCS without authorisation, you could define:
For example, if you want to use the S3 high-level interface to access public data from GCS without authorization, you could define:

````julia
struct AnonymousGCS <:AbstractAWSConfig end
struct AnonymousGCS <: AbstractAWSConfig end
struct NoCredentials end

AWS.region(aws::AnonymousGCS) = "" # No region
AWS.credentials(aws::AnonymousGCS) = NoCredentials() # No credentials
AWS.check_credentials(c::NoCredentials) = c # Skip credentials check
Expand All @@ -141,7 +144,10 @@ function AWS.generate_service_url(aws::AnonymousGCS, service::String, resource::
service == "s3" || throw(ArgumentError("Can only handle s3 requests to GCS"))
return string("https://storage.googleapis.com.", resource)
end
AWS.global_aws_config(AnonymousGCS())

with_aws_config(AnonymousGCS()) do
...
end
````

which skips some of the signature and credentials checking and modifies the generation of the endpoint url.
Expand Down Expand Up @@ -175,7 +181,10 @@ function AWS.generate_service_url(aws::MinioConfig, service::String, resource::S
service == "s3" || throw(ArgumentError("Can only handle s3 requests to Minio"))
return string(aws.endpoint, resource)
end
AWS.global_aws_config(MinioConfig("http://127.0.0.1:9000", "aregion", SimpleCredentials("minio", "minio123", "")))

with_aws_config(MinioConfig("http://127.0.0.1:9000", "aregion", SimpleCredentials("minio", "minio123", ""))) do
...
end
````

Now we are ready to use AWS.jl to do S3-compatible requests to a minio server.
Expand Down
47 changes: 7 additions & 40 deletions src/AWS.jl
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -10,6 +10,7 @@ using Mocking
using OrderedCollections: LittleDict, OrderedDict
using Random
using SHA
using ScopedValues: ScopedValues, ScopedValue, @with
using Sockets
using URIs
using UUIDs: UUIDs
Expand All @@ -19,7 +20,8 @@ export @service
export _merge
export AbstractAWSConfig, AWSConfig, AWSExceptions, AWSServices, Request
export IMDS
export assume_role, generate_service_url, global_aws_config, set_user_agent
export current_aws_config, default_aws_config, with_aws_config
export assume_role, generate_service_url, set_user_agent
export sign!, sign_aws2!, sign_aws4!
export JSONService, RestJSONService, RestXMLService, QueryService, set_features

Expand Down Expand Up @@ -65,41 +67,6 @@ Base.@kwdef struct FeatureSet
use_response_type::Bool = false
end

"""
global_aws_config()

Retrieve the global AWS configuration.
If one is not set, create one with default configuration options.

# Keywords
- `kwargs...`: AWSConfig kwargs to be passed along if the global configuration is not already set

# Returns
- `AWSConfig`: The global AWS configuration
"""
function global_aws_config(; kwargs...)
if !isassigned(aws_config) || !isempty(kwargs)
aws_config[] = AWSConfig(; kwargs...)
end

return aws_config[]
end

"""
global_aws_config(config::AbstractAWSConfig)

Set the global AWSConfig.

# Arguments
- `config::AWSConfig`: The AWSConfig to set in the global state

# Returns
- `AWSConfig`: Global AWSConfig
"""
function global_aws_config(config::AbstractAWSConfig)
return aws_config[] = config
end

"""
set_user_agent(new_user_agent::String)

Expand Down Expand Up @@ -317,7 +284,7 @@ function (service::RestXMLService)(
request_method::String,
request_uri::String,
args::AbstractDict{String,<:Any}=Dict{String,Any}();
aws_config::AbstractAWSConfig=global_aws_config(),
aws_config::AbstractAWSConfig=current_aws_config(),
feature_set::FeatureSet=FeatureSet(),
)
feature_set.use_response_type && _delete_legacy_response_kw_args!(args)
Expand Down Expand Up @@ -374,7 +341,7 @@ Perform a Query request to AWS.
function (service::QueryService)(
operation::String,
args::AbstractDict{String,<:Any}=Dict{String,Any}();
aws_config::AbstractAWSConfig=global_aws_config(),
aws_config::AbstractAWSConfig=current_aws_config(),
feature_set::FeatureSet=FeatureSet(),
)
feature_set.use_response_type && _delete_legacy_response_kw_args!(args)
Expand Down Expand Up @@ -421,7 +388,7 @@ Perform a JSON request to AWS.
function (service::JSONService)(
operation::String,
args::AbstractDict{String,<:Any}=Dict{String,Any}();
aws_config::AbstractAWSConfig=global_aws_config(),
aws_config::AbstractAWSConfig=current_aws_config(),
feature_set::FeatureSet=FeatureSet(),
)
feature_set.use_response_type && _delete_legacy_response_kw_args!(args)
Expand Down Expand Up @@ -468,7 +435,7 @@ function (service::RestJSONService)(
request_method::String,
request_uri::String,
args::AbstractDict{String,<:Any}=Dict{String,String}();
aws_config::AbstractAWSConfig=global_aws_config(),
aws_config::AbstractAWSConfig=current_aws_config(),
feature_set::FeatureSet=FeatureSet(),
)
feature_set.use_response_type && _delete_legacy_response_kw_args!(args)
Expand Down
74 changes: 74 additions & 0 deletions src/AWSConfig.jl
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -92,3 +92,77 @@ function _update_creds!(aws_config::AWSConfig)

return creds
end

const _DEFAULT_AWS_CONFIG = Ref{AbstractAWSConfig}()
const _DEFAULT_AWS_CONFIG_LOCK = ReentrantLock()
const _SCOPED_AWS_CONFIG = ScopedValue{AbstractAWSConfig}()

"""
default_aws_config(config::AbstractAWSConfig) -> Nothing

Sets the default AWS configuration to use when calling
[`current_aws_config()`](@ref current_aws_config) outside of the scope of any
[`with_aws_config`](@ref) calls.
"""
function default_aws_config(config::AbstractAWSConfig)
@lock _DEFAULT_AWS_CONFIG_LOCK begin
_DEFAULT_AWS_CONFIG[] = config
end
return nothing
end

"""
default_aws_config() -> AbstractAWSConfig

Gets the default AWS configuration to use when calling
[`current_aws_config()`](@ref current_aws_config) outside of the scope of any
[`with_aws_config`](@ref) calls. If no configuration has been set with
`default_aws_config(::AbstractAWSConfig)` then the AWS configuration will be loaded using
`AWSConfig()`.

Users will typically want to use [`current_aws_config()](@ref current_aws_config) instead.
"""
function default_aws_config()
# Load an initial default AWS configuration only if a user hasn't already loaded one.
# We'll make use of a lock to avoid race conditions but only lock when necessary to
# avoid unnecessary overhead.
if !isassigned(_DEFAULT_AWS_CONFIG)
@lock _DEFAULT_AWS_CONFIG_LOCK begin
if !isassigned(_DEFAULT_AWS_CONFIG)
_DEFAULT_AWS_CONFIG[] = @mock AWSConfig()
end
end
end

return _DEFAULT_AWS_CONFIG[]
end

"""
current_aws_config() -> AbstractAWSConfig

Retrieve the current AWS configuration set within the current scope. If no configuration has
been set using [`with_aws_config`](@ref) then the default AWS configuration will be used.
"""
function current_aws_config()
#! format: off
return @something(
ScopedValues.get(_SCOPED_AWS_CONFIG),
@mock(default_aws_config()),
)
#! format: on
end

"""
with_aws_config(f, config::AbstractAWSConfig) -> Any

Executes the function `f` where the result of `current_aws_config()` returns `config`.
Nesting of `with_aws_config` calls is supported.

The user provided function `f` takes no parameters. The `with_aws_config` function returns
the result return from `f`.
"""
function with_aws_config(f, config::AbstractAWSConfig)
return @with _SCOPED_AWS_CONFIG => config begin
f()
end
end
Comment on lines +164 to +168
Copy link
Contributor

@kleinschmidt kleinschmidt Jul 15, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

add a method that passes kwargs to AwsConfig like global_aws_config does?

Suggested change
function with_aws_config(f, config::AbstractAWSConfig)
return @with _SCOPED_AWS_CONFIG => config begin
f()
end
end
function with_aws_config(f, config::AbstractAWSConfig)
return @with _SCOPED_AWS_CONFIG => config begin
f()
end
end
with_aws_config(f; kwargs...) = with_aws_config(f, AwsConfig(; kwargs...))

Copy link
Member Author

@omus omus Jul 16, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm opposed to this as I think constructing an AWSConfig to pass in isn't much more to write and deprecating the kwarg version of global_aws_config was a pain. Additionally, this makes alternate AbstractAWSConfig implementations feel less like second class citizens.

36 changes: 18 additions & 18 deletions src/api_generation/high_level.jl
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -152,28 +152,28 @@ function _generate_high_level_definition(

if required_keys && (idempotent || headers)
return """
$formatted_function_name($(join(req_keys, ", ")); aws_config::AbstractAWSConfig=global_aws_config()) = $service_name(\"$method\", \"$request_uri\", $params_headers_str; aws_config=aws_config, feature_set=SERVICE_FEATURE_SET)
$formatted_function_name($(join(req_keys, ", ")), params::AbstractDict{String}; aws_config::AbstractAWSConfig=global_aws_config()) = $service_name(\"$method\", \"$request_uri\", Dict{String, Any}(mergewith(_merge, $params_headers_str, params)); aws_config=aws_config, feature_set=SERVICE_FEATURE_SET)
$formatted_function_name($(join(req_keys, ", ")); aws_config::AbstractAWSConfig=current_aws_config()) = $service_name(\"$method\", \"$request_uri\", $params_headers_str; aws_config=aws_config, feature_set=SERVICE_FEATURE_SET)
$formatted_function_name($(join(req_keys, ", ")), params::AbstractDict{String}; aws_config::AbstractAWSConfig=current_aws_config()) = $service_name(\"$method\", \"$request_uri\", Dict{String, Any}(mergewith(_merge, $params_headers_str, params)); aws_config=aws_config, feature_set=SERVICE_FEATURE_SET)
"""
elseif !required_keys && (idempotent || headers)
return """
$formatted_function_name(; aws_config::AbstractAWSConfig=global_aws_config()) = $service_name(\"$method\", \"$request_uri\", $params_headers_str; aws_config=aws_config, feature_set=SERVICE_FEATURE_SET)
$formatted_function_name(params::AbstractDict{String}; aws_config::AbstractAWSConfig=global_aws_config()) = $service_name(\"$method\", \"$request_uri\", Dict{String, Any}(mergewith(_merge, $params_headers_str, params)); aws_config=aws_config, feature_set=SERVICE_FEATURE_SET)
$formatted_function_name(; aws_config::AbstractAWSConfig=current_aws_config()) = $service_name(\"$method\", \"$request_uri\", $params_headers_str; aws_config=aws_config, feature_set=SERVICE_FEATURE_SET)
$formatted_function_name(params::AbstractDict{String}; aws_config::AbstractAWSConfig=current_aws_config()) = $service_name(\"$method\", \"$request_uri\", Dict{String, Any}(mergewith(_merge, $params_headers_str, params)); aws_config=aws_config, feature_set=SERVICE_FEATURE_SET)
"""
elseif required_keys && !isempty(req_kv)
return """
$formatted_function_name($(join(req_keys, ", ")); aws_config::AbstractAWSConfig=global_aws_config()) = $service_name(\"$method\", \"$request_uri\", $req_str); aws_config=aws_config, feature_set=SERVICE_FEATURE_SET)
$formatted_function_name($(join(req_keys, ", ")), params::AbstractDict{String}; aws_config::AbstractAWSConfig=global_aws_config()) = $service_name(\"$method\", \"$request_uri\", Dict{String, Any}(mergewith(_merge, $req_str), params)); aws_config=aws_config, feature_set=SERVICE_FEATURE_SET)
$formatted_function_name($(join(req_keys, ", ")); aws_config::AbstractAWSConfig=current_aws_config()) = $service_name(\"$method\", \"$request_uri\", $req_str); aws_config=aws_config, feature_set=SERVICE_FEATURE_SET)
$formatted_function_name($(join(req_keys, ", ")), params::AbstractDict{String}; aws_config::AbstractAWSConfig=current_aws_config()) = $service_name(\"$method\", \"$request_uri\", Dict{String, Any}(mergewith(_merge, $req_str), params)); aws_config=aws_config, feature_set=SERVICE_FEATURE_SET)
"""
elseif required_keys
return """
$formatted_function_name($(join(req_keys, ", ")); aws_config::AbstractAWSConfig=global_aws_config()) = $service_name(\"$method\", \"$request_uri\"; aws_config=aws_config, feature_set=SERVICE_FEATURE_SET)
$formatted_function_name($(join(req_keys, ", ")), params::AbstractDict{String}; aws_config::AbstractAWSConfig=global_aws_config()) = $service_name(\"$method\", \"$request_uri\", params; aws_config=aws_config, feature_set=SERVICE_FEATURE_SET)
$formatted_function_name($(join(req_keys, ", ")); aws_config::AbstractAWSConfig=current_aws_config()) = $service_name(\"$method\", \"$request_uri\"; aws_config=aws_config, feature_set=SERVICE_FEATURE_SET)
$formatted_function_name($(join(req_keys, ", ")), params::AbstractDict{String}; aws_config::AbstractAWSConfig=current_aws_config()) = $service_name(\"$method\", \"$request_uri\", params; aws_config=aws_config, feature_set=SERVICE_FEATURE_SET)
"""
else
return """
$formatted_function_name(; aws_config::AbstractAWSConfig=global_aws_config()) = $service_name(\"$method\", \"$request_uri\"; aws_config=aws_config, feature_set=SERVICE_FEATURE_SET)
$formatted_function_name(params::AbstractDict{String}; aws_config::AbstractAWSConfig=global_aws_config()) = $service_name(\"$method\", \"$request_uri\", params; aws_config=aws_config, feature_set=SERVICE_FEATURE_SET)
$formatted_function_name(; aws_config::AbstractAWSConfig=current_aws_config()) = $service_name(\"$method\", \"$request_uri\"; aws_config=aws_config, feature_set=SERVICE_FEATURE_SET)
$formatted_function_name(params::AbstractDict{String}; aws_config::AbstractAWSConfig=current_aws_config()) = $service_name(\"$method\", \"$request_uri\", params; aws_config=aws_config, feature_set=SERVICE_FEATURE_SET)
"""
end
end
Expand Down Expand Up @@ -201,23 +201,23 @@ function _generate_high_level_definition(

if required && idempotent
return """
$formatted_function_name($(join(req_keys, ", ")); aws_config::AbstractAWSConfig=global_aws_config()) = $service_name(\"$function_name\", Dict{String, Any}($(join(req_kv, ", ")), $(join(idempotent_kv, ", "))); aws_config=aws_config, feature_set=SERVICE_FEATURE_SET)
$formatted_function_name($(join(req_keys, ", ")), params::AbstractDict{String}; aws_config::AbstractAWSConfig=global_aws_config()) = $service_name(\"$function_name\", Dict{String, Any}(mergewith(_merge, Dict{String, Any}($(join(req_kv, ", ")), $(join(idempotent_kv, ", "))), params)); aws_config=aws_config, feature_set=SERVICE_FEATURE_SET)
$formatted_function_name($(join(req_keys, ", ")); aws_config::AbstractAWSConfig=current_aws_config()) = $service_name(\"$function_name\", Dict{String, Any}($(join(req_kv, ", ")), $(join(idempotent_kv, ", "))); aws_config=aws_config, feature_set=SERVICE_FEATURE_SET)
$formatted_function_name($(join(req_keys, ", ")), params::AbstractDict{String}; aws_config::AbstractAWSConfig=current_aws_config()) = $service_name(\"$function_name\", Dict{String, Any}(mergewith(_merge, Dict{String, Any}($(join(req_kv, ", ")), $(join(idempotent_kv, ", "))), params)); aws_config=aws_config, feature_set=SERVICE_FEATURE_SET)
"""
elseif required
return """
$formatted_function_name($(join(req_keys, ", ")); aws_config::AbstractAWSConfig=global_aws_config()) = $service_name(\"$function_name\", Dict{String, Any}($(join(req_kv, ", "))); aws_config=aws_config, feature_set=SERVICE_FEATURE_SET)
$formatted_function_name($(join(req_keys, ", ")), params::AbstractDict{String}; aws_config::AbstractAWSConfig=global_aws_config()) = $service_name(\"$function_name\", Dict{String, Any}(mergewith(_merge, Dict{String, Any}($(join(req_kv, ", "))), params)); aws_config=aws_config, feature_set=SERVICE_FEATURE_SET)
$formatted_function_name($(join(req_keys, ", ")); aws_config::AbstractAWSConfig=current_aws_config()) = $service_name(\"$function_name\", Dict{String, Any}($(join(req_kv, ", "))); aws_config=aws_config, feature_set=SERVICE_FEATURE_SET)
$formatted_function_name($(join(req_keys, ", ")), params::AbstractDict{String}; aws_config::AbstractAWSConfig=current_aws_config()) = $service_name(\"$function_name\", Dict{String, Any}(mergewith(_merge, Dict{String, Any}($(join(req_kv, ", "))), params)); aws_config=aws_config, feature_set=SERVICE_FEATURE_SET)
"""
elseif idempotent
return """
$formatted_function_name(; aws_config::AbstractAWSConfig=global_aws_config()) = $service_name(\"$function_name\", Dict{String, Any}($(join(idempotent_kv, ", "))); aws_config=aws_config, feature_set=SERVICE_FEATURE_SET)
$formatted_function_name(params::AbstractDict{String}; aws_config::AbstractAWSConfig=global_aws_config()) = $service_name(\"$function_name\", Dict{String, Any}(mergewith(_merge, Dict{String, Any}($(join(idempotent_kv, ", "))), params)); aws_config=aws_config, feature_set=SERVICE_FEATURE_SET)
$formatted_function_name(; aws_config::AbstractAWSConfig=current_aws_config()) = $service_name(\"$function_name\", Dict{String, Any}($(join(idempotent_kv, ", "))); aws_config=aws_config, feature_set=SERVICE_FEATURE_SET)
$formatted_function_name(params::AbstractDict{String}; aws_config::AbstractAWSConfig=current_aws_config()) = $service_name(\"$function_name\", Dict{String, Any}(mergewith(_merge, Dict{String, Any}($(join(idempotent_kv, ", "))), params)); aws_config=aws_config, feature_set=SERVICE_FEATURE_SET)
"""
else
return """
$formatted_function_name(; aws_config::AbstractAWSConfig=global_aws_config()) = $service_name(\"$function_name\"; aws_config=aws_config, feature_set=SERVICE_FEATURE_SET)
$formatted_function_name(params::AbstractDict{String}; aws_config::AbstractAWSConfig=global_aws_config()) = $service_name(\"$function_name\", params; aws_config=aws_config, feature_set=SERVICE_FEATURE_SET)
$formatted_function_name(; aws_config::AbstractAWSConfig=current_aws_config()) = $service_name(\"$function_name\"; aws_config=aws_config, feature_set=SERVICE_FEATURE_SET)
$formatted_function_name(params::AbstractDict{String}; aws_config::AbstractAWSConfig=current_aws_config()) = $service_name(\"$function_name\", params; aws_config=aws_config, feature_set=SERVICE_FEATURE_SET)
"""
end
end
Expand Down
Loading
Loading