Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Is NVD api host in throubles? #7004

Closed
limkinZero opened this issue Oct 1, 2024 · 1 comment
Closed

Is NVD api host in throubles? #7004

limkinZero opened this issue Oct 1, 2024 · 1 comment
Labels

Comments

@limkinZero
Copy link

limkinZero commented Oct 1, 2024

Our ci jobs are not finishing because they exceed the timeout of one hour. The task of updating the NVD database (for the first time) does not finish. In one hour, only 20%. Do you know if there is a problem?

INFO] NVD API has 264,240 records in this update
[INFO] Downloaded 10,000/264,240 (4%)
[WARNING] Retrying request /rest/json/cves/2.0?resultsPerPage=2000&startIndex=10000 : 2 time
[INFO] Downloaded 20,000/264,240 (8%)
[WARNING] Retrying request /rest/json/cves/2.0?resultsPerPage=2000&startIndex=18000 : 2 time
[WARNING] Retrying request /rest/json/cves/2.0?resultsPerPage=2000&startIndex=32000 : 2 time
[WARNING] Retrying request /rest/json/cves/2.0?resultsPerPage=2000&startIndex=36000 : 2 time
[WARNING] Retrying request /rest/json/cves/2.0?resultsPerPage=2000&startIndex=18000 : 3 time
[WARNING] Retrying request /rest/json/cves/2.0?resultsPerPage=2000&startIndex=32000 : 3 time
[WARNING] Retrying request /rest/json/cves/2.0?resultsPerPage=2000&startIndex=38000 : 2 time
[WARNING] Retrying request /rest/json/cves/2.0?resultsPerPage=2000&startIndex=36000 : 3 time
[INFO] Downloaded 30,000/264,240 (11%)
[WARNING] Retrying request /rest/json/cves/2.0?resultsPerPage=2000&startIndex=32000 : 4 time
[WARNING] NVD API request failures are occurring; retrying request for the 5 time
[WARNING] Retrying request /rest/json/cves/2.0?resultsPerPage=2000&startIndex=38000 : 3 time
[WARNING] Retrying request /rest/json/cves/2.0?resultsPerPage=2000&startIndex=36000 : 4 time
[WARNING] NVD API request failures are occurring; retrying request for the 5 time
[WARNING] NVD API request failures are occurring; retrying request for the 5 time
[WARNING] NVD API request failures are occurring; retrying request for the 6 time
[WARNING] Retrying request /rest/json/cves/2.0?resultsPerPage=2000&startIndex=38000 : 4 time
[WARNING] NVD API request failures are occurring; retrying request for the 5 time
[WARNING] NVD API request failures are occurring; retrying request for the 5 time
[WARNING] NVD API request failures are occurring; retrying request for the 6 time
[WARNING] NVD API request failures are occurring; retrying request for the 6 time
[WARNING] NVD API request failures are occurring; retrying request for the 7 time
[WARNING] NVD API request failures are occurring; retrying request for the 6 time
[WARNING] NVD API request failures are occurring; retrying request for the 7 time
[WARNING] NVD API request failures are occurring; retrying request for the 5 time
[WARNING] NVD API request failures are occurring; retrying request for the 6 time
[WARNING] NVD API request failures are occurring; retrying request for the 7 time
[WARNING] NVD API request failures are occurring; retrying request for the 8 time
[WARNING] Retrying request /rest/json/cves/2.0?resultsPerPage=2000&startIndex=34000 : 2 time
[WARNING] NVD API request failures are occurring; retrying request for the 6 time
[WARNING] NVD API request failures are occurring; retrying request for the 7 time
[WARNING] NVD API request failures are occurring; retrying request for the 7 time
[WARNING] NVD API request failures are occurring; retrying request for the 8 time
[WARNING] NVD API request failures are occurring; retrying request for the 8 time
[WARNING] NVD API request failures are occurring; retrying request for the 9 time
[WARNING] Retrying request /rest/json/cves/2.0?resultsPerPage=2000&startIndex=34000 : 3 time
[WARNING] NVD API request failures are occurring; retrying request for the 7 time
[WARNING] NVD API request failures are occurring; retrying request for the 8 time
[WARNING] NVD API request failures are occurring; retrying request for the 8 time
[WARNING] NVD API request failures are occurring; retrying request for the 9 time
[WARNING] NVD API request failures are occurring; retrying request for the 9 time
[WARNING] NVD API request failures are occurring; retrying request for the 10 time
[WARNING] NVD API request failures are occurring; retrying request for the 9 time
[WARNING] NVD API request failures are occurring; retrying request for the 10 time
[WARNING] Retrying request /rest/json/cves/2.0?resultsPerPage=2000&startIndex=34000 : 4 time
[WARNING] NVD API request failures are occurring; retrying request for the 5 time
[WARNING] NVD API request failures are occurring; retrying request for the 8 time
[WARNING] NVD API request failures are occurring; retrying request for the 9 time
[WARNING] NVD API request failures are occurring; retrying request for the 10 time
[WARNING] NVD API request failures are occurring; retrying request for the 11 time
@aikebah
Copy link
Collaborator

aikebah commented Oct 2, 2024

No issues seen on my side. We have a job that runs (with API key) every 4 hours to take in NVD API updates that are stored in the cveDB on the data folder on a persistent share on our CI environment.

Your logs indicate that you don't have such a persistent share, as it has the full 260k+ entries to download. You should cache the data folder of dependency-check across runs in order to play nicely with NVD rather than abusing their service to reload full dataset on each scan.

See also what NVD themselves have to say about using their API responsibly at https://nvd.nist.gov/general/news/API-Key-Announcement

@aikebah aikebah closed this as completed Oct 20, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests

2 participants