bindings: Add native automatic re-export feature #59859
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Keno
force-pushed
the
kf/autoreexport
branch
2 times, most recently
from
049803d to
23ca145
Compare
Keno
added a commit
that referenced
this pull request
Oct 30, 2025
This is a first WIP implementation for providing the underlying mechanism for a #54905. I haven't fully designed out what I want the final thing to look like, but I've thought about it enough that I think we need some hands-on examples for further design exploration, which is what this PR is designed to support. The core idea here is that if you specify in your package ``` [compact] julia = "0.14" MyDependency = "0.3,0.4" ``` Then this mechanism will ask `MyDependency` to provide you a view of its API that is compatible with its 0.3 version, even if 0.4 is installed. The objective of this mechanism is that downstreams of widely used packages (including `Base` and standard libraries) can be upgraded more gradually by allowing some portion of the packages in project to use the new 0.4 API set, while packages that have not yet upgraded can continue to use the 0.3 API set. Currently, whenever a widely used package changes an API there's a big frenzied rush to update downstreams, which just isn't sustainable as the ecosystem grows. In other languages, problems like these are solved by allowing multiple versions of the same package to be loaded. However, in julia, this is not particularly feasible by default because packages may have generic functions that need to be extended and unified and which break if there are multiple copies of such resources. That said, for simple packages, this mechanism could be used to emulate the multiple-version-loading strategy on an opt-in basis. The way that this works is that `loading` gains an additional layer of indirection that is provided the `compat` specification of the loading package. Packages can opt into this by providing a ``` function _get_versioned_api(compat) end ``` function. Where the default is equivalent to `_get_versioned_api(compat) = @__MODULE__`. The loader makes no further assumptions on either the structure of `compat` or how the package processes it. This is intentional to allow evolution without impacting the core loading logic (of course Pkg also looks at the structure of compat, as would the `_get_versioned_api` implementation in Base, so there are some constraints). That said, the envisioned usage of this is that we create a standard (in the sense of being widely used, not in the sense of it being a stdlib) package to help package authors define the APIs of their packages more precisely and with version information. This package would then create a set of modules under the hood that describe these APIs to the sytem and would set up `_get_versioned_api` appropriately (i.e. the mechanism is not intended to be used directly in most cases). To actually make this useful, we will likely need some additional features in the binding system, in particular: 1. #59859 and a few variants thereon to make the API modueles look more transparent. 2. A mechanism to intercept extension of generic function overloads in order to be able to provide compatibility (Design/PR for this forthcoming). Note that this PR itself is WIP, it is not yet fully complete and there is also a Pkg.jl side of this required to put compat info into the manifest.
Keno
added a commit
that referenced
this pull request
Oct 30, 2025
This is a first WIP implementation for providing the underlying mechanism for a #54905. I haven't fully designed out what I want the final thing to look like, but I've thought about it enough that I think we need some hands-on examples for further design exploration, which is what this PR is designed to support. The core idea here is that if you specify in your package ``` [compact] julia = "0.14" MyDependency = "0.3,0.4" ``` Then this mechanism will ask `MyDependency` to provide you a view of its API that is compatible with its 0.3 version, even if 0.4 is installed. The objective of this mechanism is that downstreams of widely used packages (including `Base` and standard libraries) can be upgraded more gradually by allowing some portion of the packages in project to use the new 0.4 API set, while packages that have not yet upgraded can continue to use the 0.3 API set. Currently, whenever a widely used package changes an API there's a big frenzied rush to update downstreams, which just isn't sustainable as the ecosystem grows. In other languages, problems like these are solved by allowing multiple versions of the same package to be loaded. However, in julia, this is not particularly feasible by default because packages may have generic functions that need to be extended and unified and which break if there are multiple copies of such resources. That said, for simple packages, this mechanism could be used to emulate the multiple-version-loading strategy on an opt-in basis. The way that this works is that `loading` gains an additional layer of indirection that is provided the `compat` specification of the loading package. Packages can opt into this by providing a ``` function _get_versioned_api(compat) end ``` function. Where the default is equivalent to `_get_versioned_api(compat) = @__MODULE__`. The loader makes no further assumptions on either the structure of `compat` or how the package processes it. This is intentional to allow evolution without impacting the core loading logic (of course Pkg also looks at the structure of compat, as would the `_get_versioned_api` implementation in Base, so there are some constraints). That said, the envisioned usage of this is that we create a standard (in the sense of being widely used, not in the sense of it being a stdlib) package to help package authors define the APIs of their packages more precisely and with version information. This package would then create a set of modules under the hood that describe these APIs to the sytem and would set up `_get_versioned_api` appropriately (i.e. the mechanism is not intended to be used directly in most cases). To actually make this useful, we will likely need some additional features in the binding system, in particular: 1. #59859 and a few variants thereon to make the API modueles look more transparent. 2. A mechanism to intercept extension of generic function overloads in order to be able to provide compatibility (Design/PR for this forthcoming). Note that this PR itself is WIP, it is not yet fully complete and there is also a Pkg.jl side of this required to put compat info into the manifest.
Keno
added a commit
that referenced
this pull request
Oct 30, 2025
This is a first WIP implementation for providing the underlying mechanism for a #54905. I haven't fully designed out what I want the final thing to look like, but I've thought about it enough that I think we need some hands-on examples for further design exploration, which is what this PR is designed to support. The core idea here is that if you specify in your package ``` [compact] julia = "0.14" MyDependency = "0.3,0.4" ``` Then this mechanism will ask `MyDependency` to provide you a view of its API that is compatible with its 0.3 version, even if 0.4 is installed. The objective of this mechanism is that downstreams of widely used packages (including `Base` and standard libraries) can be upgraded more gradually by allowing some portion of the packages in project to use the new 0.4 API set, while packages that have not yet upgraded can continue to use the 0.3 API set. Currently, whenever a widely used package changes an API there's a big frenzied rush to update downstreams, which just isn't sustainable as the ecosystem grows. In other languages, problems like these are solved by allowing multiple versions of the same package to be loaded. However, in julia, this is not particularly feasible by default because packages may have generic functions that need to be extended and unified and which break if there are multiple copies of such resources. That said, for simple packages, this mechanism could be used to emulate the multiple-version-loading strategy on an opt-in basis. The way that this works is that `loading` gains an additional layer of indirection that is provided the `compat` specification of the loading package. Packages can opt into this by providing a ``` function _get_versioned_api(compat) end ``` function. Where the default is equivalent to `_get_versioned_api(compat) = @__MODULE__`. The loader makes no further assumptions on either the structure of `compat` or how the package processes it. This is intentional to allow evolution without impacting the core loading logic (of course Pkg also looks at the structure of compat, as would the `_get_versioned_api` implementation in Base, so there are some constraints). That said, the envisioned usage of this is that we create a standard (in the sense of being widely used, not in the sense of it being a stdlib) package to help package authors define the APIs of their packages more precisely and with version information. This package would then create a set of modules under the hood that describe these APIs to the sytem and would set up `_get_versioned_api` appropriately (i.e. the mechanism is not intended to be used directly in most cases). To actually make this useful, we will likely need some additional features in the binding system, in particular: 1. #59859 and a few variants thereon to make the API modueles look more transparent. 2. A mechanism to intercept extension of generic function overloads in order to be able to provide compatibility (Design/PR for this forthcoming). Note that this PR itself is WIP, it is not yet fully complete and there is also a Pkg.jl side of this required to put compat info into the manifest.
This implements a native version of automatic re-export, similar to the macro from Reexport.jl. Doing this natively in the binding system has two key advantages: 1. It properly participates in binding resolution and world ages. If a new binding is Revise'd into a re-exported module, this will now propagate. 2. The re-exported bindings are allocated lazily, improving performance. An accessor for this functionality is provided as `@Base.Experimental.reexport`. However, the only supported argument to this macro is single `using` statement (unlike the Reexport.jl version, which has more syntax). It is my expectation that Reexport.jl will be updated to make use of the underlying funtionality here directly, the base version of the macro is mostly for testing. There are a few design warts here - in particular, we inherit the module name exporting behavior (JuliaLang/Reexport.jl#39). However, I think that would be best addressed by not exporting the module name itself from the modules but rather introducing the module name as an additional binding on `using` statements. However, this would be potentially breaking (even if the effect is unlikely to be seen in practice), so I'm not proposing it here. The Reexport.jl version of the macro can do whatever it needs to, including creating an explicit non-exported binding to suppress the automatic re-export for this if desired. Partially written by Claude Code.
qinsoon
added a commit
to mmtk/mmtk-julia
that referenced
this pull request
Nov 6, 2025
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This implements a native version of automatic re-export, similar to the macro from Reexport.jl. Doing this natively in the binding system has two key advantages:
It properly participates in binding resolution and world ages. If a new binding is Revise'd into a re-exported module, this will now propagate.
The re-exported bindings are allocated lazily, improving performance.
An accessor for this functionality is provided as
@Base.Experimental.reexport. However, the only supported argument to this macro is singleusingstatement (unlike the Reexport.jl version, which has more syntax). It is my expectation that Reexport.jl will be updated to make use of the underlying functionality here directly, the base version of the macro is mostly for testing.There are a few design warts here - in particular, we inherit the module name exporting behavior
(JuliaLang/Reexport.jl#39).
However, I think that would be best addressed by not exporting the module name itself from the modules but rather introducing the module name as an additional binding on
usingstatements.However, this would be potentially breaking (even if the effect is unlikely to be seen in practice), so I'm not proposing it here. The Reexport.jl version of the macro can do whatever it needs to, including creating an explicit non-exported binding to suppress the automatic re-export for this if desired.
Partially written by Claude Code.