Metadata / labels document and cleanup?

[kdvolder]

I'm trying to make use of the metadata produced by the buildpacks in the labels of the image that it produces.

I presume that it is possible to obtain information such as whether a given dependency/jar was included in the image and if so what version. This is definitely useful information and I want to know it!

However...

1. the format of the metadata needs to be clearly documented so that potential consumers of the data can
   a) understand how to parse the data
   b) rely on this parsing / format / structure to remain stable in the future (i.e. the documentation of the metadata format is to represent a contract of sorts that consumers of the data can rely on.

2. I think there may be a bit too much metadata being attached. I think this because when I use 'docker inspect' on a buildpacked container the result is a file large enough to break some editors. The file I have is 500k in size. Granted, this is 'manageble' if handled with care, but parsing that data is still costly (memory and CPU). And some tools cannot handle it at all, for example gedit linux text editor freezes up as soon as I try to search for text in this file). So I question whether all this data is really needed/useful. (Hard to say now as I don't fully understand yet what is actually there. Some of it though seems to be the complete textual documentation for spring boot metadata properties, I think we probably do not really need all that documentation embedded in the metadata).

[nebhale]

1. @ekcasey is working hard on documenting this (much of it is in the spec, but should be surfaced in a more user-friendly way). I'll have her run the documentation past you once it's published.
2. The amount of metadata attached is by design. By adding all possible Boot configurations for a given application (as described by Boot itself; it's the standard Boot configuration metadata), the goal is to enable GUIs that offer collections of properties (complete with their documentation) to users.

[dmikusa]

@kdvolder

An update:

1. I believe you're referring to the bill of materials (BOM). It is accessible with pack inspect apps/maven --bom or looking at the label directly. This format is currently custom and I do not believe documented, so here's some info.

The format is as such:

• pack inspect will return JSON, the label is JSON as well
• pack inspect will return a dict with two keys: remote and local. They indicate the same information, but show the information for the label on your local Docker daemon vs on the remote registry. Often they are the same but could differ if there is a newer image in the registry, for example.
• The label, or remote/local entries are a JSON array. The array contains a dict, one for each layer, that is contributed to the image.
• Each layer contributes the following info:
  • name -> the BOM entry's name
  • metadata -> a custom set of information, specific to each buildpack (I can't document all of these here, but the format is pretty simple and hopefully obvious)
  • buildpacks -> the buildpack that contributed the BOM entry

I don't anticipate this format to change until we implement this RFC. The RFC is focused on getting a standard format available for the bill of materials. Final comments are I believe tomorrow, and once it's finalized we can look at implementing it. The format will be standard, like SPDX or CycloneDX, so after this is implemented the format won't change & will be documented.

2. We have taken some steps to address item Can't run simple app on heroku due to memory calculator #2. See Apps with huge amount of files possible lead to K8s node DoS spring-boot#80 (comment) for details. There is still some information in the labels though.

If there are any labels you're still concerned about let me know which ones and I can get some more details.

Hope that helps!

[kdvolder]

This format is currently custom and I do not believe documented

Yeah, so that's what this issue is about. For the format to be documented.

metadata -> a custom set of information, specific to each buildpack (I can't document all of these here, but the format is pretty simple and hopefully obvious)

This is really the most interesting and crucial part. E.g. for my use case it was about discovering dependencies in metadata from the java buildpack.

but the format is pretty simple and hopefully obvious

It really isn't obvious, and even if it was, the fact that these things are wholly undocumented essentially raises the questions:

• how much can you rely on the format remaining unchanged and compatible with whatever use you are making of it?
• how confident can you be that the format doesn't vary in some breaking way depending on factors you may not be aware of. E.g. for some 'hypothetical' parameters that could impact the data: maven vs gradle project, project using custom layering vs default structure, project using a non-default base image / builder etc. project using war vs jar packaging etc. etc. So how much variability do these parameters introduce into the part of the metadata that one is interested in?

Without documentations spelling out what the format is, it is hard / impossible to answer such questions with any confidence. Also even if you think you have it 'figured out' you are still running the risk it may change tomorrow because there really isn't any kind of contract a consumer of this data can rely upon.

[dmikusa]

I get what you're saying, it should be documented and explained. That makes the format an API and something that won't change out from under you. I understand that's important for anyone that wants to integrate with the tools.

Being completely transparent and as direct as possible, the present format isn't going to be documented or be any sort of official or guaranteed format. It'll continue to exist in its present format for the time being. We have no plans to change it until, as I mentioned before, the RFC that's coming that will prescribe a BOM format based on industry-wide standards like SPDX and CycloneDX. We'll be implementing that RFC once it is formalized.

This RFC is also important because it will unify the format across all of the buildpacks. Right now, we can only control what the Java buildpacks are contributing to the BOM & its format. Other buildpacks and the stack make contributions as well. Those formats are out of our control, at least until this RFC is implemented.

I believe that will address your concerns, but if not, please let me know. I can also leave this issue open if you like until that RFC has been implemented & we roll it out. Just let me know.

[kdvolder]

Whatever you want to do is fine, re keeping the ticket open or not. This ticket is old and sort of forgot about it. At the moment I have no real issue. That may change if something breaks in our tools and I have to look at fixing it :-). In that case I may be back here with some questions.

[dmikusa]

I'll keep it open & I will update it when we start to and when we implement the transition for buildpacks/rfcs#166. If anyone is interested in this feature set, feel free to watch this issue.

It also seems reasonable that this RFC implementation/change will trigger a major version bump, since it's changing existing established behaviors. So you can watch for that as well.

[dmikusa]

The Paketo buildpacks have implemented the new Buildpacks RFC for SBOM.

Here are some notes on the transition:

1. The lifecycle was not implemented to support both the old style & new style SBOM formats. If you specify both, the lifecycle will error.
2. Due to # 1, Paketo Java buildpacks cannot offer a transition period where we support both formats. We had hoped to be able to do this, but it's just not possible with the lifecyle.
3. Due to # 2 and this being a breaking change, we have bumped the major version on all of the Paketo Java buildpacks.
4. To use the new SBOM format in the lifecycle, you need the following:
   • Platform API 0.8+
   • Buildpack API 0.7+
5. At the moment of writing this, there is no release of pack that includes platform 0.8. There is an RC you can use and a release should be available soon. The Spring Boot integration only at present supports platform API 0.4, and there is a plan to upgrade it to 0.8 for Spring Boot 2.7.
6. With the major version bump of Paketo Java buildpacks, we have bumped the buildpack API to 0.7.
7. If you have a compatible platform and you are using the latest major buildpack versions, you will get the new-style SBOM output. This is bundled in the container image under the /layers/sbom/.
8. For bundled dependencies & helper dependencies, we are presently outputting Syft JSON SBOM format. We'll be adding CycloneDX format in the future.
9. For application dependencies, we are presently outputting Syft JSON & CycloneDx JSON SBOM format.

Here's an example:

10. If you are using an older platform, you cannot get new-style SBOM output. You can continue to get old-style SBOM output by keeping your buildpack API set to 0.6. You can do this by sticking with paketo-buildpacks/java v5.21.1 or by packaging your own version of the buildpacks with the API set to 0.6 (it needs to be set for every buildpack that generates SBOM output).

I'm going to keep this issue open for a little while, so feel free to post questions/feedback here, or reach out in our Slack channel.

[kdvolder]

Okay so I tried the command but all I get is a bom which is 'null' for both 'local' and 'remote'.

$ pack inspect-image test --bom
{
  "remote": null,
  "local": null
}

It looks some stuff did break in our tools. It is related to determining whether a given image contains a specific Java dependency. The code we had for this used to look at the labels directly. And that code broke. Also looking at all the labels on the image I do not see any label there that still has this information. So I wonder how we are supposed to access that information now. If you can provide some pointers that would be appreciated.

[kdvolder]

Note: the image was build using pack cli and the builder paketobuildpacks/builder:full set as default builder. The project we built was a very vanilla spring-boot web app created using spring initializer.