Add --incremental option to hex.registry build

[aj-foster]

Hello folks 👋🏼

This PR is primarily for discussion purposes. The code is for demonstration, but I welcome all feedback.

The mix hex.registry build command is great for managing local package registries. Thank you for implementing it. Occasionally we want to add a package (or package version) to a registry without regenerating the entire registry. For example, if we build a package .tar in a CI job and want to publish it to a local registry, it might be difficult to download all of the existing tarballs prior to building the registry.

Perhaps there is a way around this. If so, great! No need for this PR. If there isn't, maybe this could help.

This PR implements an --incremental option for mix hex.registry build. When this option is enabled, it adds new packages to a registry without removing any existing packages. Thus, if you have a single tarball in PUBLIC_DIR/tarballs, then a single package will be added (or updated). All existing packages remain. The CI job could then re-upload the names, versions, and packages/X files along with the new tarball.

At the time of writing there are two commits with two variations of an implementation. The first keeps the incremental logic separate in a separate function. The second attempts to integrate it with the existing function. I welcome feedback on everything. 🙂

Thank you for everything you do in the community!

[wojtekmach]

Thank you for the PR!

I think a feature like this is very welcome, especially for really large registries.

Instead of an mix hex.registry build --incremental that assumes a public/tarball directory with "incomplete" contents, what do you think about a mix hex.registry add /path/to/foo-1.0.0.tar? I feel like it would be more explicit way of handling what seems to be most common use case here - publishing a single release. (On the other hand, --incremental might be more useful for registry syncing perhaps.) With add, for completeness, we'd most likely also add mix hex.registry remove|retire eventually but there is no rush.

cc @ericmj

[aj-foster]

That makes sense. Would the add command copy the specified tarball into the correct directory? If I'm understanding the workflow correctly, one use-case might be...

• Create an initial repository with any number of packages using hex.registry build
• Copy the full public directory to a provider, if appropriate

Then, later in CI (for example)...

• Copy the names, versions, and packages/foo files to the CI environment
• Add a newly-generated tarball with hex.registry add /path/to/foo-1.0.0.tar
• Sync the names, versions, packages/foo, and tarballs/foo-1.0.0.tar to the provider

I agree that an --incremental build may still be useful, but it's definitely a more narrow use-case.

[wojtekmach]

Would the add command copy the specified tarball into the correct directory?

Yes, exactly! To be precise, it would copy the tarball to repositories tarballs/ directory and rebuild the registry. When you add multiple tarballs, it would be inefficient to add them one at a time, but I don't think we need to worry about it yet. I suppose we could later extend the command to mix hex.registry add TARBALL1 TARBALL2 ... but yeah, let's wait for a use case.

I agree that an --incremental build may still be useful, but it's definitely a more narrow use-case.

Right, that was my thinking as well. Let's hold off on this flag then, we can always add it later.

Sorry for late reply!

[aj-foster]

Closing this in favor of #931 for now. 🙂