new Args class for relocateable configurations

[lefou]

This proposal add details how configuration could be made more relocateable in context of

• Make out/ folder contents (more) reproducible and filesystem layout agnostic #3660

Intro

If we assume, that all paths represented via os.Path or PathRef are handled by Mill in a relocateable ways (as implemented in PR #6031), we still have the issue that various other String-typed confguration may also contain paths.

Where else may absolute path be a problem?

1. The most prominent occurences are tool options like javacOptions or scalacOptions, which may refer to additional plugins or agent jars via a argument string like -Xplugin path/to/plugin.jar.

2. Another cause is stored absolute paths in the produced output of tools. I think, for this kind of issue, there is no generic solution. Either a tool supports some reproducibility options (e.g. bnd -reproducible=true), supports some virtual environment configuration (like zinc's VitualFile) or some post-processing is needed. Sometimes it also helps to sanitize inputs/outputs like MANIFEST.MF files and remove timestamps and exact tools versions (if they are not significant).

Here I try to address 1.

The problem

Mill's JSON serialization is based on statically typing. To reliably encode and map paths, we need to encode them in types like os.Path. Seq[String] isn't well suited to hold mixed typed data and can therefore not be used to encode path semantics in a type-safe and robust way.

Proposal

I propose a new data type for configs: Args, which should be used instead of the Seq[String] type we use currently. That type is designed to specifically handle paths as part of otherwise stringly typed configuration data.

The proposed new case classes are:

• Args - a Seq-like container representing all config parameters. Args can hold Arg and ArgGroups. It can be easily converted to Seq[String] and supports bidirectional JSON serialization

• Arg - a single config parameter like -deprecated or -f.
   
  An Arg can hold a String, a os.Path or multiple of these representing parts. Those parts will effectively be merged into a single arg (once converted to a String), but can be handled separately in serialization, allowing us to apply path-mappings without falling back to string-manipulations. This allows us to represent parameters like -Xplugin=<file> as Arg("-Xplugin=", os.home / "libs/plugin.jar")

• ArgGroup (kind like Args, but we don't want to support recursive nesting) - a group of Args that are contextually bound and should not used separately, e.g. -source and 17. By binding these options, we can easily override some settings without any special parsing, e.g. to override a default to -source 21, we could just search for groups starting with -source and replace it.

Raw Example

val pluginPath: os.Path   = ???
val sources: Seq[os.Path] = ???
Args(
  Arg("java"),
  ArgGroup(Arg("-source"), Arg("17")),
  Arg("-deprecation"),
  Arg("-Xplugin=", pluginPath"),
  sources.map(Arg(_))
)

Syntactic sugar

How Args can be defined in Scala isn't finally set in stone and depends largely on experimentation and tradeoffs of easy-writing vs. easy-reading.

I haven't finished my experimentation, but I think we can come up with enough Scala-like convenience to make reading and writing easy enough and a good replacement for Seq[String].
We could use implicit conversion, to use String or os.Path where we expect an Arg. I could imagine to also implicitly convert tuples to ArgGroups, but that might be not trivial without a lot of boilerplate code. And hopefully, we can use a StringContext to easily write multi-part args via string interpolation, but I haven't tested that yet.

val pluginPath: os.Path   = ???
val sources: Seq[os.Path] = ???
Args(
  "java",
  ("-source", "17"),
  "-deprecation",
  arg"-Xplugin=${pluginPath}"
  sources
)

Compatibility

Using the new type in existing tasks would be an binary breaking change, so converting existing tasks to it would mean, we need to wait for Mill 2.0.

On the other hand, the new type can hold the same data as Seq[String], so it can be immediately used internally and in new API.

[lefou]

I added a draft implementation of the Args API to show, that it properly handles paths in configuration args, when used with a Mill build based on PR #6031

Here is the gist: https://gist.github.com/lefou/4ab9e44c119559ff5e31e1e761d552a4