Making Cybersecurity Fun with a Capture the Flag Event, Part 2

In Part 1 of our Capture the Flag (“CTF”) writeup, we explored the social and business value of running a hacking competition. Now, let’s take a behind-the-scenes look at the technologies we used to run the event (and if you’re feeling brave, try to hack your own way to victory).

Sudo: a fickle beast

Our resident Site Reliability Engineer, Bob Micheletto, has a penchant for finding flaws in poorly written sudo rules. He initially wrote 21 sudo-based challenges, but to keep things simple, we pared them down to 10 for our contest. (Note: you can try all 21 here.)

Several of these challenges were derived from exploitable sudo rules I've encountered in the wild throughout my career.

Bob Micheletto

Site Reliability Engineer

Network Ninja

We decided that participants in the event would:

• SSH to a preconfigured server,
• Examine the /etc/sudoers file, and
• Attempt to gain root access by exploiting those sudo rules.

If and when they got root, hackers would run a special /root/ctf.sh command, which would send a secret hash and automatically update the leaderboard.

Technology to run the event

Server setup

We considered a few hosting options:

1. Docker containers with screen recording & manually submitting solutions, which we ruled out because they’d be too difficult to support. (Note: if you want to attempt the challenges yourself, you will be using Docker.) ❌
2. Shared AWS ec2 instances, which were nixed because it’d be too easy to accidentally disable the system for all - plus folks might leave clues to challenges laying around. ❌
3. Single AWS t2.nano instances for all, to be refreshed every day. It cost less than $5 for a server month, and we’d only keep them up for a couple weeks anyway. Perfect. ✅

TL;DR - folks would ssh to their own t2.nano instance each day, and figure out the challenge from there.

Spawning the servers

Each challenge server was provisioned using AWS CloudFormation and configured using our agentless Puppet configuration. Our cloud-init script fetched the Puppet configuration from code-commit and applied the appropriate challenge tailored for a specific user based on the EC2 instance tags.

User accounts & authentication

We used a Puppet module to create users on our servers, and user assignment is done through Hiera, so we generated a bunch of configuration files for the challenge servers that put only User X, on Server X. “We didn’t want anyone to be tempted to log into another user’s challenge server,” Bob explains.

We added our users public keys using the same module that adds our users. The snag was, we were going to be deleting and recreating these servers 10 times a piece. To avoid users running into SSH warnings about changing host keys, we decided to preserve the keys and provision them through Puppet.

Time limits

To enforce the time limits for challenges - with automatic logouts after a specified time - we used a shutdown trick from Unixdaemon.net. “We needed a way to keep the challenges short and sweet for our contestants,” Bob says. “Adding time limits - 10 minutes for easy challenges, and 20 minutes for medium ones - proved ideal.”

Here’s the Puppet template we used to create the timer events on login:

#!/bin/sh

if [ "$PAM_USER" = '<%= @user %>' ] ; then
    # Start the events
    echo /usr/local/bin/shutdown_timer.sh | /usr/bin/at now
    echo /usr/local/bin/hint.sh | /usr/bin/at now
fi

exit 0

Note: we still wanted the ability to login to these servers with administrative user, so we check the PAM_USER environment variable - and only kickoff the events if the challenge user logs in.

Here’s the Puppet template file we used to generate the shutdown:

#!/bin/sh

# Shutdown Delay
sleep <%= @shutdown_delay %>

cat /usr/local/shutdown_message.txt | /usr/bin/wall

/usr/sbin/shutdown -h now

Note: the duration of shutdown_delay was different depending on the difficulty of the challenge (10 minutes for easy ones, 20 minutes for medium).

Scoring

For automated scoring - with a leaderboard contestants could check at any time - we wrote a script that:

• Sent out a statsd metric, and
• Sent a verification email (using AWS SNS) with a secret string for verification, unique per user, per challenge. Without this, anyone could “hack” the leaderboard by spoofing statsd messages - and that’s not the type of hacking we’re looking for here.

We then built a leaderboard using AWS managed Grafana, and allowed our users to login to the leaderboard using their usual SSO logins. Here’s a glimpse of the final standings:

“Automating the scoring process took a bit of wrangling, but saved us a lot of time and manual work in the end,” Bob concludes.

Providing hints

Automated hints appeared for contestants halfway through each challenge using wall. The hint timer was set at the same time as the shutdown timer (on user login). “Frustration would ruin the fun, but we also didn’t want to give away too much,” Bob says. “Our hints provided just enough info to point folks in the right direction.”

Here’s the Puppet template we used to send the hint:

#!/bin/sh

sleep <%= @hint_delay %>

cat /usr/local/hint.txt | /usr/bin/wall

exit 0

The hint.txt file was also configured via Puppet, and was specific to each challenge.

Network Ninja branding

A special flourish for contestants was this custom Network Ninja ANSI MOTD that appeared on login.

Support, documentation, & fun provided via Slack

Each morning at 9am ET, we’d announce the daily challenge in our company Slack at #ctf2022.

Contestants had 24 hours to complete each challenge. For educational purposes, we also provided reference solutions for the prior day’s challenge in the channel.

Some challenges had multiple possible solutions, and it was cool to see people figure out their own solutions I hadn’t anticipated, too.

Bob Micheletto

Site Reliability Engineer

Network Ninja

“I accidentally trashed the sudoers file”

Here are a smattering of contest highlights from Slack. Note to the reader: don’t do the following.

Hacker creativity reigned supreme.

After several lockout notices, this warning attempted to keep people from harming their own shells…

…But not everyone listened.

Seriously, it’s really easy to break things with sudo/sudoers!

When stuck, extra help was available upon request.

In contrast, others gloated about their hacking prowess, in song form.

And now, a final song to send us off. We can’t wait for #ctf2023!

CTF FTW

Throughout the competition, we had fun, collaborated together across product teams, and learned some things about security along the way. Be sure to check out Part 1 of our CTF event writeup if you haven’t already, and if you want to participate in the competition next time, come work with us.