npm audit vs auditjs

Mike Hoskins — Wed, 11 Mar 2020 23:32:37 +0000

[Full disclosure: I work for Sonatype, the company who offers OSS Index and AuditJS as free tools. We do this to give back to the OpenSource community, help raise awareness through easier access to security data, and garner interest in the extended feature set of commercial DevSecOps tooling we provide.]

Awhile back I wrote a blog post after a colleague shared a new JavaScript auditing tool called AuditJS. I wanted to update that based on more time with the tool, particularly since a new version was recently released!

AuditJS is a free tool leveraging Sonatype's OSS Index. OSSI exposes a ReST API aggregating several security vulnerability feeds including CVE, CWE and NVD. OSSI continues to evolve by adding more data sources and benefits from ongoing curation of existing feeds. The effort required to mine so much data is conveniently abstracted away as it should be by a good tool -- setup AuditJS to reap the benefits!

The 4.x release brought a lot of bug fixes and usability enhancements based on community feedback... Installation is quick and easy, if you want to run AuditJS as a NPM script just npm i auditjs -D to get started. I prefer running it via npx auditjs ossi.

You can use it as a one-off CLI or easily integrate with your CI/CD pipelines. It's possible to integrate with the commercial IQ Server (benefits from additional curation including both human and machine intelligence), but it is completely free to use with OSSI. You might want to register for a free account, but even that is not required. The one benefit of an account is removing the rate-limit which can affect larger projects. I haven't hit rate-limit issues in my typically-sized NodeJS projects. You can even submit vulnerability reports via an awesome Git-based process.

Let's see what it looks like, and talk about a couple things which might surprise you when comparing to npm audit:

➜ npx auditjs ossi
 ________   ___  ___   ________   ___   _________       ___   ________
|\   __  \ |\  \|\  \ |\   ___ \ |\  \ |\___   ___\    |\  \ |\   ____\
\ \  \|\  \\ \  \\\  \\ \  \_|\ \\ \  \\|___ \  \_|    \ \  \\ \  \___|_
 \ \   __  \\ \  \\\  \\ \  \ \\ \\ \  \    \ \  \   __ \ \  \\ \_____  \
  \ \  \ \  \\ \  \\\  \\ \  \_\\ \\ \  \    \ \  \ |\  \\_\  \\|____|\  \
   \ \__\ \__\\ \_______\\ \_______\\ \__\    \ \__\\ \________\ ____\_\  \
    \|__|\|__| \|_______| \|_______| \|__|     \|__| \|________||\_________\
                                                                \|_________|

  _      _                       _   _
 /_)    /_`_  _  _ _/_   _  _   (/  /_`_._  _   _/ _
/_)/_/ ._//_// //_|/ /_//_//_' (_X /  ///_'/ //_/_\
   _/                _//

  AuditJS version: 4.0.10

✔ Starting application
✔ Getting coordinates for Sonatype OSS Index
✔ Auditing your application with Sonatype OSS Index
✔ Submitting coordinates to Sonatype OSS Index
✔ Reticulating splines
✔ Removing whitelisted vulnerabilities

  Sonabot here, beep boop beep boop, here are your Sonatype OSS Index results:
  Total dependencies audited: 224

[1/224] - pkg:npm/@nodelib/fs.scandir@2.1.3 - No vulnerabilities found!
[2/224] - pkg:npm/@nodelib/fs.stat@2.0.3 - No vulnerabilities found!
[3/224] - pkg:npm/@nodelib/fs.walk@1.2.4 - No vulnerabilities found!
[4/224] - pkg:npm/@sendgrid/client@6.5.3 - No vulnerabilities found!
[5/224] - pkg:npm/@sendgrid/helpers@6.5.3 - No vulnerabilities found!
[6/224] - pkg:npm/@sendgrid/mail@6.5.4 - No vulnerabilities found!
[7/224] - pkg:npm/@testim/chrome-version@1.0.7 - No vulnerabilities found!
[8/224] - pkg:npm/@types/caseless@0.12.2 - No vulnerabilities found!
[9/224] - pkg:npm/@types/events@3.0.0 - No vulnerabilities found!

# Result list trimmed...

Dependency lists can obviously be long in NodeJS projects (understatement of the decade?), but the important thing is the reference to Sonatype's OSS Index (yay it's working!) as well as the ability to whitelist.

Let's say you are alerted about a vulnerability, but know it doesn’t affect you because you aren’t using the vulnerable method -- you can whitelist that! Just pass in a whitelist file containing the OSS Index IDs (the only required field, but you can add others for clarity):

➜ cat my-whitelist.json
{
  "ignore": [
    { "id": "long-oss-index-guid", "reason": "I accept the risk!" },
    { "id": "another-oss-index-guid", "reason": "We totally got this!" }
  ]
}

➜ npx auditjs ossi --whitelist my-whitelist.json

The ideal is obviously to fix all the things, but this puts control in the hands of the developer and is especially useful in larger projects or CI/CD pipelines where the risk of lower severity issues is understood and potentially annoying people or breaking builds while waiting on upstream fixes.

The other thing I want to call out is the dependency count. Above we see 224 dependencies were audited. By default, AuditJS only scans production dependencies. This is similar to --only=prod with NPM, but for AuditJS we need --dev to force scanning everything. Let's compare:

# Could also use some --json | jq fu!
➜ npx auditjs ossi 2>&1|grep -e '^\['|wc
     224    1568   14428

➜ npx auditjs ossi --dev 2>&1|grep -e '^\['|wc
     885    6195   58537

# Nothing too crazy here:
➜ jq .devDependencies <package.json
{
  "acorn": "^7.1.1",
  "acorn-jsx": "^5.2.0",
  "ajv": "^6.12.0",
  "auditjs": "^4.0.10",
  "eslint": "^6.8.0",
  "eslint-config-airbnb": "^18.0.1",
  "eslint-config-airbnb-base": "^14.0.0",
  "eslint-config-prettier": "^6.10.0",
  "eslint-plugin-import": "^2.20.1",
  "eslint-plugin-jsx-a11y": "^6.2.3",
  "eslint-plugin-node": "^11.0.0",
  "eslint-plugin-prettier": "^3.1.2",
  "eslint-plugin-promise": "^4.2.1",
  "eslint-plugin-react": "^7.19.0",
  "eslint-plugin-react-hooks": "^2.5.0",
  "jest": "^25.1.0",
  "nodemon": "^2.0.2",
  "prettier": "^1.19.1",
  "supertest": "^4.0.2"
}

Even a modest set of devDependencies requires a lot of additional scanning (~75% more in this case). AuditJS scans production dependencies by default, as these are what will get shipped with your built product, making it easier to understand the risk profile. You can add devDependencies in if you want!

Aside from scanning behavior, another place AuditJS attempts to be more efficient is in reporting. NPM will often over-inflate vulnerability reports (to be fair, we might call this erring on the side of caution). Often, with NPM, you'll see output similar to the following:

➜ npm audit

...

added 1405 packages from 1327 contributors and audited 896097 packages in 26.484s
found 18 moderate severity vulnerabilities

896,097 packages?!? Or is it 1405? What gives -- I don't write efficient code (I likes me some cowsay), but I don't see that much stuff in node_packages...

Let's break that down:

➜ npm ls --parseable | wc
    1057    1057   88307

➜ npm ls --parseable --only=prod | wc
     231     231   18007

231 is a lot closer to AuditJS' default behavior... Where did those extra packages come from? The devil is in the details, and this can lead to confusion when comparing tools... AuditJS de-dupes, so you still get warned about any critical vulnerabilities despite potentially looking like it did less work. Here's an example of how NPM reports on the same package multiple times:

➜ npm ls --parseable|grep -E 'ms$'
src/thing/node_modules/log4js/node_modules/ms
src/thing/node_modules/streamroller/node_modules/ms
src/thing/node_modules/rewire/node_modules/ms
src/thing/node_modules/eslint/node_modules/ms
src/thing/node_modules/ms
src/thing/node_modules/send/node_modules/debug/node_modules/ms
src/thing/node_modules/send/node_modules/ms
...

ms is indeed used in all these different places, but in total only three versions of it exist in my project. AuditJS only reports on the distinct coordinates it finds (2.0.0, 2.1.1, 2.1.2), whereas npm ls (and the associated npm audit commands) count multiple times if something is duplicated.

In the arms race that is security, it's nice to see new tools which help developers stay ahead of the bad guys are rapidly evolving. For your next JavaScript project, take a moment to test drive some npm audit alternatives!

Forem: Mike Hoskins

npm audit vs auditjs