Improve CVE entry management in OSV

[hogo6002]

The current OSV structure combines vulnerability data from different resources (e.g., NVD, Alpine, Debian) into a single CVE entry based on shared CVE IDs. This approach leads to overly large and difficult to maintain CVE entries. With Ubuntu also publishing its security tracker data to OSV.dev using CVE IDs, this issue will likely worsen.

We probably need a better solution for managing CVE entries. One idea is to add a source-specific prefix or suffix to the CVE ID, creating separate entries for each data source. For example, Alpine-CVE-2024-0001 and Debian-CVE-2024-0001 would be displayed as two distinct records on OSV.dev.

[dodys]

As someone that does not have a view of the whole osv.dev machinery, I think the main question is, why would you want to combine CVE entries?

[hogo6002]

why would you want to combine CVE entries?

The initial reason was that Alpine SecDB uses CVE IDs to publish their vulnerabilities. So, we created a tool to combine their data into our existing CVEs, as long as they use the same vulnerability ID. Then, we wanted to add Debian security tracker data into our OSV to better support container scanning, so we continued to combine everything into our CVE entries to keep it consistent, as the Debian security tracker also uses CVE IDs.

However, the problem now is that we continue to combine more and more data into a single record, increasing the difficulty of our maintenance. So, we are considering some better changes to this approach.

[dodys]

why would you want to combine CVE entries?

The initial reason was that Alpine SecDB uses CVE IDs to publish their vulnerabilities. So, we created a tool to combine their data into our existing CVEs, as long as they use the same vulnerability ID. Then, we wanted to add Debian security tracker data into our OSV to better support container scanning, so we continued to combine everything into our CVE entries to keep it consistent, as the Debian security tracker also uses CVE IDs.

However, the problem now is that we continue to combine more and more data into a single record, increasing the difficulty of our maintenance. So, we are considering some better changes to this approach.

From your first message, the thing that is not clear to me is, if you are not going to combine CVEs anymore, then why would you need to still add the prefixes (e.g. Debian-CVE-2024-0001) ? Aren't those completely separate buckets?

[oliverchang]

why would you want to combine CVE entries?

The initial reason was that Alpine SecDB uses CVE IDs to publish their vulnerabilities. So, we created a tool to combine their data into our existing CVEs, as long as they use the same vulnerability ID. Then, we wanted to add Debian security tracker data into our OSV to better support container scanning, so we continued to combine everything into our CVE entries to keep it consistent, as the Debian security tracker also uses CVE IDs.
However, the problem now is that we continue to combine more and more data into a single record, increasing the difficulty of our maintenance. So, we are considering some better changes to this approach.

From your first message, the thing that is not clear to me is, if you are not going to combine CVEs anymore, then why would you need to still add the prefixes (e.g. Debian-CVE-2024-0001) ? Aren't those completely separate buckets?

For OSV, ID prefixes must be unique, and be associated with a home database. Debian security tracker, Alpine etc didn't have a unique ID before, so we were combining them into a single CVE record. If we don't combine them, then they would need their own prefix.

[dodys]

why would you want to combine CVE entries?

The initial reason was that Alpine SecDB uses CVE IDs to publish their vulnerabilities. So, we created a tool to combine their data into our existing CVEs, as long as they use the same vulnerability ID. Then, we wanted to add Debian security tracker data into our OSV to better support container scanning, so we continued to combine everything into our CVE entries to keep it consistent, as the Debian security tracker also uses CVE IDs.
However, the problem now is that we continue to combine more and more data into a single record, increasing the difficulty of our maintenance. So, we are considering some better changes to this approach.

From your first message, the thing that is not clear to me is, if you are not going to combine CVEs anymore, then why would you need to still add the prefixes (e.g. Debian-CVE-2024-0001) ? Aren't those completely separate buckets?

For OSV, ID prefixes must be unique, and be associated with a home database. Debian security tracker, Alpine etc didn't have a unique ID before, so we were combining them into a single CVE record. If we don't combine them, then they would need their own prefix.

Thanks for clarifying it, and would the prefix also be necessary for the filename itself? or could we have a filename CVE-2024-0001.json and inside of it id = "UBTU-CVE-2024-0001?

[hogo6002]

Thanks for clarifying it, and would the prefix also be necessary for the filename itself? or could we have a filename CVE-2024-0001.json and inside of it id = "UBTU-CVE-2024-0001?

We do prefer the filename to match the ID name to keep it consistent for other ecosystems. Are there any concerns for Ubuntu in changing the filename?

[dodys]

Thanks for clarifying it, and would the prefix also be necessary for the filename itself? or could we have a filename CVE-2024-0001.json and inside of it id = "UBTU-CVE-2024-0001?

We do prefer the filename to match the ID name to keep it consistent for other ecosystems. Are there any concerns for Ubuntu in changing the filename?

No concerns, just confirming what needs to be done as it will be a mass operation.

The only thing to confirm is which should be the prefix, Ubuntu-, UBUNTU-, UBTU-?

[hogo6002]

The only thing to confirm is which should be the prefix, Ubuntu-, UBUNTU-, UBTU-?

Thanks for confirming! We prefer UBUNTU- to align with our current naming schema. Same as https://osv.dev/vulnerability/CURL-CVE-2024-7264

[hogo6002]

Hey @dodys, I have two questions about Ubuntu CVE OSV records (ubuntu-security-notices/osv).
First, I'm curious why some affected packages lack the ubuntu_priority field. This field is from the vulnerability level and should be consistent across all affected packages. But I've noticed that some packages are missing this field while others within the same vulnerability have it. Example: https://osv.dev/vulnerability/CVE-2024-21131

Second question is about the status field. I can't find it in the Ubuntu OSV records, not sure if it's an important one. But I'm wondering if there's a filtering mechanism that determines which records Ubuntu sends to OSV (like not sending all records with a status of DNE or not-affected ?).

[dodys]

Hey @dodys, I have two questions about Ubuntu CVE OSV records (ubuntu-security-notices/osv). First, I'm curious why some affected packages lack the ubuntu_priority field. This field is from the vulnerability level and should be consistent across all affected packages. But I've noticed that some packages are missing this field while others within the same vulnerability have it. Example: https://osv.dev/vulnerability/CVE-2024-21131

Indeed, for now we only added the ecosystem_specific part (including ubuntu_priority) when the CVE was fixed. In the next batch of update to our scripts, we plan to add the ecosystem_specific to all.

Second question is about the status field. I can't find it in the Ubuntu OSV records, not sure if it's an important one. But I'm wondering if there's a filtering mechanism that determines which records Ubuntu sends to OSV (like not sending all records with a status of DNE or not-affected ?).

Since OSV has the affected field, which is for only packages that are vulnerable to such "vulnerability" (CVE) or "set of vulnerabilities" (USN), for CVEs, we only list the ones that are affected or are still pending investigation. That means that the following status are always included:
needed -> means vulnerable/affected
needs-triage -> means that it needs investigation, therefore we consider it vulnerable until investigation is done
deferred -> means that we tried to investigate but we still don't have enough information to do so, therefore it is vulnerable until confirmed
pending -> means that the package is vulnerable and a fix is on its way
released -> means that we patched it in such release

Some status are more complex and will depend on other factors:
not-affected (<version>) -> if the <version> exists in that specific Ubuntu release, we consider it similar to released, if the <version> is an old one, e.g. the first version to contain the fix and not vulnerable anymore, then we don't add it since that Ubuntu release is not vulnerable at all.
ignored -> it will only be included for releases that are still "alive", as ignored can be used for different reasons.

And some status we just don't include:
DNE -> package doesn't exist in such Ubuntu release
active -> even though it is in the docs we still don't use that status.

Does that help clarify?

[hogo6002]

Does that help clarify?

Thanks Eduardo! It's very detailed and helpful!

Having all the ubuntu_priority info should be sufficient for us, especially considering you've already helped us filter out DNE and some not-affected ones.

[dodys]

I forgot to ask one thing Holly
For the USN data, should we change the related field to refer to the UBUNTU-CVE-... or should we keep it as CVE-...?

[hogo6002]

I forgot to ask one thing Holly For the USN data, should we change the related field to refer to the UBUNTU-CVE-... or should we keep it as CVE-...?

Are you referring to data from USN notes such as USN-6969-1? I think for this type of data, we can just keep its original USN- prefix, no change needed. Just for Ubuntu CVEs, we can add UBUNTU-CVE- as its prefix.

Misread the question (didn't see the related word 😅), please refer to Oliver's answer

[oliverchang]

Hmm, I think @dodys is asking about what to put inside related for USN- entries? I think under our schema definitions, it would make the most sense to:

1. Put UBUNTU-CVE- in aliases, if it exists, given that they're both Ubuntu-specific advisories talking about the same vulnerability.
2. Put CVE- in related.