A mirror hack should not be catastrophic

This is a continuation of my thoughts on the csclub.uwaterloo.ca mirror hack.

This could have been catastrophic. An attacker could have modified the sync scripts to compromise all the files we served from our mirror.

Yes, they could control all files served by the mirror, but this presumes that upstream projects trusted the mirror operators in the first place.¹ While this may be the case for mirroring their own files, any third party project which lists the CSC mirror as an official mirror need not, and should not, naively trust that they will serve the same files they received.

If projects did, becoming a popular mirror and then swapping out files with malicious ones would be big business. And while it takes some effort to run a mirror, I would bet it’s far simpler than many other methods of distributing malware.

Instead, projects sign the files (commonly application packages) they serve with known asymmetric encryption keys that users verify. For OS distributions like Arch or Alpine, users get a copy of the public key(s) in their initial download of the ISO from a first-party source, and then their respective package managers (pacman and apk) verify that all future downloads are signed by the respective private key(s).

Of the 61 projects mirrored, over 75% have signatures attached, so malicious files would be quickly detected and the mirror operators informed.² There would be no catastrophe. This security is the very purpose of package signing.

Several things had gone wrong here. The “mirror” user didn’t have a password. The default settings of ProFTPD were too insecure for our use case, and the documentation is incredibly hard-to-read.³ We didn’t prevent logins via FTP. We didn’t prevent unnecessary access to home directories. We didn’t check our FTP logs often, or have an automated way to do it. If even one of these factors had been mitigated, the breach that we finally detected wouldn’t have occurred. Syscom would eventually switch over to vsftpd (very-secure FTP daemon, a more secure implementation with much more readable documentation) instead, for a simpler configuration to avoid further mishaps.⁴

I think the primary lessons here, of using software you personally understand, and minimizing your attack surface by choosing the most minimal, secure application for the job, are key. After all, as a system administrator — some of whom are even software engineers — responsible for important infrastructure, is not knowing the software you run a good excuse for getting pwned?

Of course I don’t mean you should perform a software audit on everything you use, or know every part of the Linux kernel, or know all applicable RFCs by heart,⁵ but you should be confident in your choice of tool for the job. If all you need to do is run a script every hour, use cron, not Ansible; if you can use a new tool or reuse an existing one, prefer the latter. And yes, there’s caveats and exceptions and difficulties in all of this, but that’s part of what makes this job so interesting — figuring out what’s best.

Thankfully, this is something the CSC sysadmins know, so I’m happy to hear security is improving, and I’ll continue to use the CSC mirror.

When asked for comment for this blog post, zseguin emphasized that it’s “important to know what’s running, where your points of entry are and where your logs are”, […]. [merenber] also mentioned that syscom “needs to be more proactive” and that “the vsftpd configuration is far simpler (only one config file)… This makes it far less likely for someone to screw it up.”

Food for Thought

The article concludes with three questions to the reader.

1. Can you identify where ProFTPD documents (or even find the source implementation) that it allows writes and logins from all users by default?

See the previous post where I also discuss the rationale behind this default.

2. What are some additional ways we could have prevented this breach, or detected it earlier, that aren’t mentioned in this post?

As analyzed in the previous post, there’s several alternatives that could have prevented the breach (or fixed it afterwards):

• Disable logging in to the mirror user.
• Add the mirror user to /etc/ftpusers.
• Run ProFTPD as the mirror user.⁶

3. If you wrote an FTP daemon, what are some guard rails you would add to prevent incidents like this?

I think the author wants readers to say “prevent all writes by default.” That’s fine, but I’d like to propose a different solution: keep the default but add a quick start guide that goes over starting the daemon and mention that all users can now upload files. Then, include a short snippet of how this can be prevented. This has the added benefit of being applicable to ProFTPD without breaking backwards compatibility.