Do-not-underestimate-credentials-leaks.md

Do not underestimate credentials leaks.

While one might think that leaking credentials is pretty much a noob mistake, and if you are feeling yourself being safe because you are an experienced developer, that’s a wrong impression leading to underestimation of the problem.

I performed a rather simple credentials search using only two methods: wget/untar/grep on npm packages and GitHub Search. The queries were pretty simple, it did not take much time, and that’s a quite obvious thing to do. Do you remember yourself laughing at «begin rsa private key» search results over GitHub and people who publish that?

So, here are the results

I obtained direct access to the following packages (publishing on npm or/and GitHub push access), you might have heard of some of them:

• Express,
• EventEmitter2,
• mime-types,
• semver,
• npm,
• fstream,
• cookie (and cookies),
• Bower,
• Component (double, via two unrelated users),
• Connect, koa, co, tar, css, gm, csrf, keygrip, jcarousel, serialport, basic-auth, lru-cache, inflight, mochify, denodeify, and a lot of other packages.

Note that some packages were exposed more than once via unrelated users.

Some packages were indirectly affected — the packages that are not affected by themselves, but that depend on affected packages, so that installing such package installs an affected package. As an example: Grunt depends on EventEmmiter2, so it was also indirectly affected. Request, Browserify, grunt, grunt-cli, cordova, forever, less, pm2, node-gyp, karma, yo, strongloop, glob, mocha, sails, and a lot of other packages were all also indirectly affected (via deps). Most of that is chained — for example, request depends on mime-types, and a lot of packages depend on request.

Taking indirectly affected (via deps) packages into an account, it could be seen that from the 15 packages people 'npm install' a lot only two were not affected.

Overall, there were around 90-135 npm passwords, around 40-80 npm tokens, around 100 GitHub tokens, and a few GitHub passwords. Those are the ones that I found, scans done by the corresponding teams most probably found more of them.

Common sources of leaks

This is not an comprehensive list of course, just a few examples:

1. Dotfiles: various things, for example:
2. .ssh private keys (duh) — if you are publishing that, you are in great trouble.
3. .npmrc (duh) — npm auth.
4. .muttrc (duh) — email auth data.
5. sublime text config (wut?) — GitHub oAuth tokens.
6. config.json (duh) — GitHub oAuth tokens, supposedly from Composer.
7. .gitconfig — (duh) — GitHub auth / oAuth tokens.
8. .netrc — (duh) passwords.
9. Per-package stuff: both inside npm packages and on public git repos (not only limited to Github):
10. .npmrc (duh) — npm auth. Always excluded from npm packages nowdays if you are using a recent 2.x/3.x npm version.
11. package.json / Gemfile / Gemfile.lock / bower.json (wut?) — people sometimes put their GitHub oAuth tokens inside them in a repo url. Ow!
12. config.gypi — saves the build process environment (wut?). Old npm clients leaking their credentials into the environment of child processes (in some cases) makes this even more dangerous.
13. travis.yml (ow) — could contain sensitive stuff like API keys (including npm logins/passwords or authTokens). Not everyone uses the encryption. They should.
14. config.json (duh) — GitHub oAuth tokens, supposedly from Composer. Also present in composer.json, .travis.composer.config.json, etc., and some configuration scripts.
15. Websites/scripts configuration:
16. DB auth data (duh), frequently in a form of a connection string.
17. GitHub oAuth tokens (wut?). People use tokens with write permissions to all their repos on their webpages, just to fetch some stuff from Github.
18. GitHub passwords (wut?). People use that in their scripts or configuration.
19. Logs.

All sort of stuff could get into logs, if something fails to take care of that. It's better to inspect logs before publishing them. Logs can save your passwords, the environment variables, http logs could save auth data, some logs can save what you type or search. Note: those are not theoretical examples, I saw all of those on GitHub. Be extremely careful when publishing logs.

Specific examples

Express, mime-types, Component, …

What: npm password, npm token.

Where: npm.

Exposed through: credentials leak through config.gypi in one package saving environment variables and npm leaking it’s credentials to the env of child processes in some cases.

Fixed: user changed password and revoked the token. On npm side — by both not leaking credentials to the env (client-side) and automatically revoking packaged tokens/passwords (server-side).

Window: December 2014 — September 2015.

npm, semver, …

What: npm password.

Where: GitHub Gist.

Exposed through: credentials leak through a published log file saving environment variables and npm leaking it’s credentials to the env of child processes in some cases.

Fixed: user changed password and revoked the token. On npm side — by both not leaking credentials to the env (client-side) and automatically revoking packaged tokens/passwords (server-side).

Window: April 2015 — October 2015.

Bower

What: GitHub oAuth token (with push access to all the repos in the org, including the website).

Where: GitHub.

Exposed through: GitHub oAuth token leak through a Gemfile in another repo.

Fixed: user revoked the token. On GitHub side — by automatically revoking publically commited tokens.

Window: April 2015 — September 2015.

EventEmitter2, Component (again), …

What: npm password and GitHub password.

Where: GitHub.

Exposed through: Password leak in dotfiles. The password was shared between npm and github.

Fixed: user changed passwords.

Window: First half of 2013 — October 2015.

JCarousel

What: GitHub password.

Where: npm.

Exposed through: Password leak through config file of an utility (that shouldn’t be included) in two packages.

Fixed: user changed passwords.

Window: April 2014 — August 2015.

---

There were a lot of other packages, I am not including all of them.

The main point of this

Being an experienced or a certified Node.js developer, being an Node.js expert consultant, having over 500 packages on npm or having yourself in the top few answerers on StackOverflow in your field, being big, being a CEO, having tens of thousands stars on GitHub or followers on Twitter, working for Intel, Google, or npm itself, studying/researching at Berkley, Harvard or MIT, having kids — nothing of that is going to secure you from credentials leakage by itself.

So, the solution for this problem is to:

1. On npm side — credentials leakage into the environment was fixed.
2. On npm and GitHub side — automatic search for their auth data in public packages/repositories at the time of publishing/committing/making a private repo public, then revoking the leaked tokens and resetting leaked passwords. This is done now, and GitHub was already doing that earlier, but their filters were misconfigured.
3. On your side — check your packages and repositories. Especially (but not limited to) dotfiles. If you share your dotfiles, review every file there.
4. If you found that your npm password has been made public, the only solution for the moment is to contact support (after resetting the password, of course). See Q/A below.
5. If you have leaked your password and/or token — review all packages and/or repos that you have access to (not only limited to your own repos). Note that push access on GitHub means that the commit that could be pushed from your account could be authored by either you or another person, so reviewing only your own commits is not enough. See Q/A below.

Check your stuff. Now.

And keep it safe from now on.

Q/A

Note: I'm not affiliated with GitHub or npm, Inc., so everything below is pretty much my personal opinion.

1. Was I affected?

   • If you did receive an email from npm/GitHub support (and your password/tokens was reset/revoked) — you were affected, if you did receive an email from me — you were also affected.

     Please, don't re-use the same password to «fool» the robot while restoring your password — this will result in your account being vulnerable. Yes, several people have done that.

   • If you didn't receive any emails and your password/tokens did not get reset/revoked — that means that your account was not found to be leaking credentials, but that does not guarantee anything. You should better check your stuff yourself.

     I targeted mainly GitHub/npm credentials, and my analysis wasn't comprehensive.

     npm performed a search for npm credentials in packages published on npmjs.org (and they are now running it for all new packages). GitHub catches tokens that are made public on GitHub.

     There is no guarantee that you (or anyone else) did not commit your email password or your db access string to either npm or GitHub. You should take care about that yourself.

2. Is my npm version fine?

   Credentials leak to the into the environment of child processes was fixed in npm v2.14.1.

   If your version is older that that (i.e. if you use Node.js v0.10 or v0.12.x lower than v0.12.8 and haven't updated npm manually) — you should update to the latest npm version in either 3.x or 2.x branch.

   Note: This only affects users that are using an account at some registry (i.e. are registered at npmjs.org).

3. I was affected to credentials leak, what should I do?

4. My npm authToken was leaked. What should I do now?

   • Revoke the affected authToken using npm logout on the machine where it was used (if that was not done for you already by npm support) or using https://npmjs.com/settings/tokens.

   • Fix the source of the leak to make sure that you don't expose your authToken again.

   • After that, review any new versions of packages that you have access to and that were published from your account.

     Note that this is not only limited to the latest version, the attacker could publish a 1.0.4 version if there are 1.0.3 and 2.0.1 in an attempt to make the interference less noticeable (especially if 1.x branch is still being used more than 2.x).

   • When in doubt — contact npm support.

5. My npm password was leaked. What should I do now?

   • Change your password, then revoke all your auth tokens using https://npmjs.com/settings/tokens.

     Revoking all the tokens is necessary for two reasons:

     • An attacker could have created a new token (using npm login) that would still be functional after a password change.
     • The complete token list is available at the above link, so an attacker could just save the whole list and use those tokens even after you change the password.

   • If the password was shared with other accounts — secure those too, by resetting the password on those accounts and performing the other necessary actions.

   • Fix the source of the leak to make sure that you don't expose your password again.

   • After that, review any new versions of packages that you have access to and that were published from your account.

     Note that this is not only limited to the latest version, the attacker could publish a 1.0.4 version if there are 1.0.3 and 2.0.1 in an attempt to make the interference less noticeable (especially if 1.x branch is still being used more than 2.x).

   • When in doubt — contact npm support.

6. My GitHub token was leaked. What should I do now?

   • Check the scopes of the leaked token. If there are no scopes — it can't do any harm.

   • If the list of scopes is not empty — revoke the affected oAuth token (if that was not done for you already by GitHub support).

   • Fix the source of the leak to make sure that you don't expose your tokens again.

   • If there was an repo (or public_repo) scope — review if any changes were made to repos (or public repos) that you have access to.

     Note that changes that were possibly made from your account are not limited to commits authored by you, they could be authored by anyone.

   • When in doubt — contact GitHub support.

7. My GitHub password was leaked. What should I do now?

   • Change your password, then revoke your oAuth tokens (if that was not done for you already by GitHub support).

   • If possible — enable 2FA.

   • If the password was shared with other accounts — secure those too, by resetting the password on those accounts and performing the other necessary actions.

   • Fix the source of the leak to make sure that you don't expose your password again.

   • After that, review if any changes were made to repos that you have access to.

     Note that changes that were possibly made from your account are not limited to commits authored by you, they could be authored by anyone.

   • When in doubt — contact GitHub support.

8. How come no one noticed this earlier?

   No idea. Perhaps no one checked.

9. Publishing that information about leaked tokens is bad, you shouldn't have done that.

   Well, not publishing this is much worse. People should be aware of dangers this brings, and should know what to do in case of credentials leaks.

   Also, people should better check what they publish, and this is an attmpt to make them more cautious.

   I'm not pointing onto specific accounts or people. Please, don't try to find and accuse specific people based on the package names that I mentioned — that's not polite and will do no good, also the access configuration was changed for some packages since then, so you will probably get wrong results.

10. What lessons should we learn here?

11. You should better watch your credentials.

12. Don't underestimate credentials leaks (well, as well as other security risks).

13. Don't make assumptions, better check. It turned out that some people weren't aware that npm packages everything in the current dir that's not ignored, some people somewhy were under an impression that it follows .git. On a side note: I saw all kinds of irrelevant stuff being packaged.

14. Plaintext passwords are bad. No, seriously. There were much more passwords leaked than tokens, and that's because people still have their password in .npmrc.

    If you have _auth or _password in your .npmrc — remove it, and perform an npm login to generate an _authToken instead.

    I opened npm/npm#9866 to address that at npm side, but there seems to be no answer yet.

15. npm lacks a way to list or revoke all the authTokens. That is bad for the reasons described above.

    I was told that this is being worked on.

    Solved now with https://npmjs.com/settings/tokens.

16. There are a lot of people with publishing access to some popular packages, and I doubt that's really needed.

    I suspect that one of the reasons for this is to include some persons into the «Contributors» list on package page.

    I was told by npm team that a more fine-grained access level control is being worked on.

17. Pretty much harm could be done by only a few leaked credentials. There are a number of extremely popular packages that pretty much everything depends upon.

    I'm not talking about npm/Express now, but about mime-types, fstream, EventEmitter2, semver, cookie, etc. Updates in those packages, even in the old branches, should not be left unnoticed.

    Perhaps some visual view of version updates (not only the latest ones, but all branches) of the top downloaded packages would be nice.

18. Why was April such a bad month for the ecosystem security?

    No idea, really. Looks like coincidence to me.

19. What about automatic tools or pre-commit hooks that scan for credentials? Should I write/use one?

    While using an automatic checker would do no direct harm, most probably it would just give you a false sense of security, but will not protect you from the actual leak.

    There are many ways how various credentials could leak, and a general purpose tool aimed to catch all kinds of credentials (even targeted exclusively towards leaks trough git repos or npm packages) would be quite complex and will inevitably be subject to both false negatives and false positives.

    Instead, you should better review stuff that you publish. That includes commit review, package contents review before publishing them, config files review, logs review before sharing them. If you have a team of several developers — it would be better to educate your devs more and make each commit go through an independent review instead of forcing your team to use some automatic tool. Also, don't forget about checking package contents.

---

Special thanks to GitHub and npm teams for taking the issue seriously and patching their server side to automatically invalidate leaked tokens/passwords when they are published on their own service (GitHub repos for GitHub oAuth tokens, npm packages for npm tokens and passwords). Note: GitHub has already been doing that, but it seems that their filter has not been catching all the cases.

---

Published: 2015-12-04.

Historic revisions (which are not in this repo): https://gist.github.com/ChALkeR/7485b79f6ef15768a309/revisions?page=2

If you have any questions to me, contact me over Gitter (@ChALkeR) or IRC (ChALkeR@freenode).

In regards to .npmrc credentials leaks, I have been sending emails to many people and companies who did that, until npm, Inc. started properly taking care of that from their side.
I received a significant bounty from one thirdparty company that have published their .npmrc file for disclosing that to them (though they did not have a public bounty program).

The vulnerability in npm itself (credentials leak to config.gypi) was not covered by any bounty reward programs.
If you want to support me so that I would be able to keep doing what I am doing, consider supporting me on Patreon.
Current supporters are listed on my fundraising page.