Allow OssAnalyser cache management

[Maxouwell]

Is your feature request related to a problem? Please describe.
Currently for NVD, we use the Single node database update strategy.
Could the OssAnalyzer cache be managed in the same way ?

Describe the solution you'd like
Make autoUpdate=false works for OssAnalyser, and a new update-oss-analyzer-only maven goal.

Describe alternatives you've considered
There seems to be no alternative ?

[chadlwilson]

There is a cache which you’ll be re-using if you save/restore the entire data directory, but it’d only have cached results for whatever components your single node updater has retrieved, and they’d expire after 24 hours. In practice this means a single node updater strategy probably won’t work efficiently with OSSIndex.

To my knowledge there is no complete dump of OSS index (it’s Sonatypes IP so I don’t see why they’d allow this) so it’s not possible to do what is done for the NVD and make it work with auto update false. Thus I don’t believe what you are asking for is technically possible without risking failures between single node updates if you add a new component version etc.

There is some more discussion at #6587 on the caching.

If you need more robust OSSIndex usage, you should probably look at paying Sonatype money :) https://ossindex.sonatype.org/doc/auth-required

[marcelstoer]

I don't understand this feature request. ossindexAnalyzerUseCache is enabled by default. Cached results expire after 24 hours.

[chadlwilson]

The difference is that with the NVD single node update strategy you download an entire db of all vulnerabilities and CPEs which can then be used to answer all possible queries for n hours afterwards.

That’s not really possible for OSSIndex, it works component by component and so a cache miss means needing to talk out to OSS index, and potentially still getting rate limited etc.

OP is asking if there is a way to have an essentially-offline cache for all of OSSIndex - which it is (to my knowledge) not.

[marcelstoer]

a way to have an essentially-offline cache for all of OSSIndex

There's none and IIRC the Sonatype terms explicitly prohibit this. So, if that's indeed what is asked there, we need to close this issue.

[chadlwilson]

In other words, when saving the data directory NVD, retireJS and KEV operate as an entire mirror of sorts. However OSSIndex is still a pull-through cache in the normal sense of ‘cache’.

I don’t use the update-only or no-update functionality so I am not sure how it works for OSSIndex, but if folks are ok with missing OSSIndex or failures on no-update usages (due to missing results) perhaps there is still an enhancement possible here? Will let the OP clarify their intent before closing.

[eu-rlarsen]

There is a cache which you’ll be re-using if you save/restore the entire data directory, but it’d only have cached results for whatever components your single node updater has retrieved, and they’d expire after 24 hours. In practice this means a single node updater strategy probably won’t work efficiently with OSSIndex.

To my knowledge there is no complete dump of OSS index (it’s Sonatypes IP so I don’t see why they’d allow this) so it’s not possible to do what is done for the NVD and make it work with auto update false. Thus I don’t believe what you are asking for is technically possible without risking failures between single node updates if you add a new component version etc.

There is some more discussion at #6587 on the caching.

If you need more robust OSSIndex usage, you should probably look at paying Sonatype money :) https://ossindex.sonatype.org/doc/auth-required

Yeah now we need to create an account at sonartype, and then set the ossIndexUsername & ossIndexPassword

and:
"Smaller teams and hobbyists can continue using Sonatype OSS Index without charge. The component-based limits mean you can scan the same components as often as you like, which enables full DevOps best practices without penalty. Usage only scales as you add more applications and developers — in other words, at true enterprise scale."

[Maxouwell]

Basically, the question boiled down to "Does OSSIndex want us to stop hammering their servers or to pay them for usage".
The first part warranted a possibility to have an extended cache and a single point of update (ala NVD), the second not so much.
Furthermore, I didn't know the OSSIndex were component-based, which kind of render the point moot.

Looking in the code at #6587 and then digging in the OSSIndex code, I can add for anyone interested that :

• The cache is in the data folder of dependency check
• There seems to be a distinct cache for each component
• The cache duration actually positioned at 24h by dependency check is specific for each component
• The file write are using a lock based strategy, so probably safe for multiple parallel run of dependency check.

Since, as far as I read, there is no details on usage limits (or paid tier) on OSSIndex at this time, will have to wait to see how it boils down in the end.

As a side note, it may be useful to make the 24h cache duration a dependency check parameter.

[chadlwilson]

As a side note, it may be useful to make the 24h cache duration a dependency check parameter.

If you raise a dedicated enhancement ticket perhaps someone will evaluate and pick it up 👍