Never use nuget.org in your CI/CD Pipeline!

Some of you may already be aware that a researcher was able to hack over 35 top tech companies like Microsoft, Apple, PayPal, etc. using a so called “software supply chain attack”. Since our old build server died we needed to rebuild a new one and coincidently we were also looking at some security aspects while building the new server. My colleague Felix did a quick proof of concept and it was shockingly easy to break our build or even inject malicious code in our build without us even noticing.

If you have a private nuget feed for closed, in-house packages, an attacker could easily “inject” a malicious package in your deployment.

The Attack Vector

So, what’s the problem having nuget.org as a source in your build pipeline?

Problem 1: Package Id and Version

As described in the link above, an attacker could quite easily “guess” a package name (package id) and the version of one of one of your packages. Since we developer are always keen to follow conventions when it comes to naming and versioning, one could guess package names as well. All an attacker needs to do is to publish a package with the same name (id) and the version you are using on nuget.org which contains custom malicious code. The default nuget.config contains nuget.org as a source and this way it’s easy to just “replace” a private package you are using with a malicious public one.

Problem 2: Source Order

When you type

nuget sources list

you will get something like this:

If you think you can simply change the order of your feeds and move the nuget.org feed at the very top to ensure it the last source to ask for a package, you are on the wrong path. A nuget restore will always ask all available sources at the same time and restores a package from the source which answers first.

Problem 3: Trusted-Signers

There’s a nuget feature called trusted-signers. It allows you to create a list of allowed authors and/or repositories. In theory this would be a great way to control which packages you accept but in reality it’s not really useful to prevent this kind of attack. The problem is, that author signatures are not required when submitting packages to nuget.org. Only a handful of authors (like Microsoft) publish packages that way. Trusting the repository is not an option because anyone could publish any package to nuget.org and effectively use the attack vector described as Problem 1.

The Solution

As I mentioned above, we are already using a private nuget feed for our closed, in-house packages (RoyalNuget). This feed is part of our Azure DevOps Project and it’s really easy to set up and configure. This private nuget feed (Azure DevOps Artifacts) can also be configured as a “proxy” for upstream nuget packages. This means, you can get to packages from nuget.org through your private feed. This feature brings some crucial advantages to remedy the problems listed above.

Once you created your feed, click on the gear icon at the top right corner:

Click on “Add upstream source” and add the Public Source “Nuget Gallery” to your Upstream Sources:

Additionally, a top level nuget.config in your repo to disable all access to nuget.org:

<?xml version="1.0" encoding="utf-8"?>
<configuration>
  <packageSources>
    <add key="nuget.org" value="https://api.nuget.org/v3/index.json" protocolVersion="3" />
  </packageSources>
  <disabledPackageSources>
    <add key="nuget.org" value="true" />
  </disabledPackageSources>
</configuration>

Since I’m a bit paranoid, I also created a hosts entry for api.nuget.org to prevent access to the public nuget feed at all time.

Alternatively you can use the trusted-signers list to only allow the private nuget feed. In my tests this didn’t work but I guess it’s a problem with the nuget version I used and might be fixed in future versions.

With this in place, an attacker can still guess and upload malicious packages to nuget.org but since you are now forced to pull all packages (private and public) through your private feed with the upstream source, it is guaranteed to be deterministic. This means, you will always get the packages from your private feed first and if they are not available there, you will get them from the upstream source (nuget.org).

Caveats

There are some situations where you may run into problems:

Hosting Packages from Upstream Sources

Sometimes you may need to host a package temporarily on your private feed. Maybe a hotfix version you got from a 3rd party vendor who also posts their packages to nuget.org. With the above setup, you cannot push a package to your private feed which is also hosted on one of your upstream sources. In this case you need to remove the upstream source, push those packages and then re-add the upstream source. After that, you will always get the packages from your private feed first.

What about newer Versions?

Microsoft recently changed the behavior in Azure Artifacts to be more secure out of the box. By default, you won’t get the newer version of such a package from the upstream source because it has already been registered with your private feed once – even if you un-list and delete the package on your private feed. This is a good thing and prevents another attack vector. For public packages you already trust, you still can allow this and make sure that newer versions are getting pulled from the upstream source again.

If you have deleted the package, it’s metadata is still on the feed and you need to make sure you see even deleted packages on your feed:

Once you located the package, click on it to get to the “Overview” of that package. There you will find an icon at the top right corner:

Toggle that switch to “On” to allow newer versions from upstream sources. Took me a while to find that option as the UX/discoverability is really bad for a setting like this.

Conclusion

Putting these measures in place doesn’t protect you from all the threats out there. Relying on external packages is always risky and you should be careful what packages you pull in. Do some vetting, report packages which are suspicious or malicious on the nuget.org package page. The measures above are relatively simple and increases security to a level where I’m much more confident. If all the packages you use are coming from nuget.org, no need to put a private feed in front of it but if you do have private packages, the above steps are the least you need to do to make sure your deployment process is not prone to this attack.