Skip to content

Conversation

G-Rath
Copy link
Collaborator

@G-Rath G-Rath commented Sep 22, 2025

This updates the local matcher to have it skip advisories that don't have at least one affected entry with a package name matching one of the packages being scanned in the current run, which can greatly reduce the peak memory usage for databases like Ubuntu (going from something like 10gb down to 1gb).

Since we cache databases based on their ecosystem only, this does mean subsequent calls to LocalMatcher#MatchVulnerabilities will not give any results for packages that were not present in the first call - while this shouldn't be a problem currently since we handle creating the VulnerabilityMatcher as part of scanning, I've added a basic guard that returns an error if the function is called with any "partial" database cached to catch this (be it on purpose or because of a bug)

This should not impact guided remediation since it explicitly loads the database before doing any work meaning this change won't help it but should also not hurt it

Resolves #2217 (again)

@G-Rath G-Rath force-pushed the skip-advisories-locally branch 2 times, most recently from 67db513 to fea59d2 Compare September 22, 2025 03:37
@codecov-commenter
Copy link

codecov-commenter commented Sep 22, 2025

Codecov Report

❌ Patch coverage is 88.88889% with 5 lines in your changes missing coverage. Please review.
✅ Project coverage is 67.07%. Comparing base (76887ec) to head (0e31c7c).

Files with missing lines Patch % Lines
...al/clients/clientimpl/localmatcher/localmatcher.go 73.68% 4 Missing and 1 partial ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main    #2241      +/-   ##
==========================================
+ Coverage   66.53%   67.07%   +0.54%     
==========================================
  Files         169      169              
  Lines       16379    16429      +50     
==========================================
+ Hits        10897    11019     +122     
- Misses       4713     4731      +18     
+ Partials      769      679      -90     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@G-Rath G-Rath changed the title perf(local): only load advisories that are about the packages being s… perf(local): only load advisories that are about the packages being scanned Sep 22, 2025
@G-Rath G-Rath force-pushed the skip-advisories-locally branch from fea59d2 to ee40413 Compare September 22, 2025 03:51
@G-Rath G-Rath force-pushed the skip-advisories-locally branch from ee40413 to 53a40b3 Compare September 22, 2025 18:56
another-rex pushed a commit that referenced this pull request Sep 23, 2025
…flag values (#2244)

I realized we were missing these while working on #2241
@G-Rath G-Rath force-pushed the skip-advisories-locally branch from 0e31c7c to 311867d Compare September 23, 2025 19:34
@G-Rath G-Rath force-pushed the skip-advisories-locally branch from 311867d to 364707c Compare September 24, 2025 00:57
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

osv-scanner in offline mode has massive memory leak
3 participants