Duplication of mbedtls-framework

[irwir]

The current directory structure requires two references and thus two copies of the framework when building mbedTLS library; the same for local git repositories.
Maybe "flat" repository structure could be considered, where each repository is directory at the same level?
Directories for mbedTLS:
[mbedtls]
[tf-psa-crypto]
[mbedtls-framework]

For TF-PSA-crypto:
[tf-psa-crypto]
[mbedtls-framework]

In this case framework might be reused for both mbedTLS and TF-PSA-crypto.

[davidhorstmann-arm]

Hi, thanks for the suggestion!

I think there are some good reasons why we have 2 submodules in that way at the moment (although I appreciate that it is a bit annoying to use).

One problem is that the framework submodule is also shared with Mbed TLS 3.6, so that limits what we can do in terms of changing the directory structure (it's an LTS, so users are relying on it to be the same). We made a submodule /framework when we released 3.6 and then we were committed to having that same structure for the lifetime of 3.6.

However, when 3.6 reaches end-of-life (in March 2027) we will have more freedom. It's possible that we'll be able to do away with the framework entirely, since we can separate it into (things that are used by crypto) and (things that are used only by TLS/X.509).

@ronald-cron-arm made most of the design decisions around the way the framework works. Ronald, do you have anything to add to this?

[gilles-peskine-arm]

We have a hard constraint that TF-PSA-Crypto should work without involving Mbed TLS, so it needs direct access to the framework. We also don't want to require users to track down dependencies, so the framework goes under the project, not separately, and likewise tf-psa-crypto goes under mbedtls. As a consequence, the nesting mbedtls/tf-psa-crypto/framework must exist.

We could avoid the duplication in several ways, but it doesn't come for free.

For background, the goal of the framework repository is to contain files that are shared between the two projects, or shared between maintained branches of one projects, such as parts of build scripts, test scripts and test data. Both projects consume files from the framework. The framework has a single branch, so at any time, Mbed TLS and TF-PSA-Crypto might use the same framework commit, or Mbed TLS might need a more recent commit, or TF-PSA-Crypto might need a more recent commit — at least we won't have the two projects wanting separate branches of the framework.

We could decide that Mbed TLS only ever consumes the framework through TF-PSA-Crypto. But that would require that every time Mbed TLS needs a framework change, we have to update TF-PSA-Crypto's framework pointer, then Mbed TLS's tf-psa-crypto pointer. So far we haven't judged this acceptable.

We could decide that when TF-PSA-Crypto is consumed by Mbed TLS, crypto's framework is not checked out and not used. This should be doable because the framework should never become incompatible with either project. But it requires a lot of work in build scripts, to make the framework location configurable. It's also beneficial only if you don't tell git to do a recursive submodule update, which we can do, but might not be applicable when Mbed TLS itself is a submodule of some other project.

There's no solution that's clearly superior to what we have now, so for the foreseeable future, the status quo wins.

[ronald-cron-arm]

There's solution that's clearly superior to what we have now, so for the foreseeable future, the status quo wins.

You mean "There's NO solution that's clearly superior to what we have now, so for the foreseeable future, the status quo wins" ?

[irwir]

We have a hard constraint that TF-PSA-Crypto should work without involving Mbed TLS, so it needs direct access to the framework. We also don't want to require users to track down dependencies, so the framework goes under the project, not separately, and likewise tf-psa-crypto goes under mbedtls. As a consequence, the nesting mbedtls/tf-psa-crypto/framework must exist.

Right now in mbedtls repository: include, library and framework are at the same level, but framework is a link.
The idea is to move contents of the current root of two repositories into a subdirectory.

The root of mbedtls repository contains only mbedtls real directory, while tf-psa-crypto and framework are links.
The root of tf-psa-crypto repository contains only tf-psa-crypto real directory, and framework is a link.
Paths in the scripts have to be rearranged, but is this really impossible without nesting?

[gilles-peskine-arm]

You mean "There's NO solution that's clearly superior to what we have now, so for the foreseeable future, the status quo wins" ?

Oops, yes.

The idea is to move contents of the current root of two repositories into a subdirectory.

That wouldn't help. You'd still have the Mbed TLS project with a framework subdirectory and a tf-psa-crypto subdirectory, and the tf-psa-crypto subdirectory would also have a framework subdirectory.

but framework is a link.

Having a link has its own problems (does Windows support symlinks well these days?), and doesn't solve anything. framework can't be a link in TF-PSA-Crypto because it has to work without Mbed TLS. If we decide that framework can be a link to tf-psa-crypto/framework in Mbed TLS, then we could just rewrite our scripts to use the path tf-psa-crypto/framework instead, and we wouldn't need a link, but we'd be in the situation that we don't want, where an Mbed TLS framework update requires a TF-PSA-Crypto framework update.

[irwir]

Having a link has its own problems

That exactly what we see now; and to make things even more troublesome - nested links.
But these links are in github. Locally there would be no links, it is three ordinary directories (local repositories) at the same level - this is the goal of the suggestion.