Making Cybersecurity Fun with a Capture the Flag Event, Part 2

Check out the technology behind our competition - and attempt the hacking challenges yourself.

In Part 1 of our Capture the Flag (“CTF”) writeup, we explored the social and business value of running a hacking competition. Now, let’s take a behind-the-scenes look at the technologies we used to run the event (and if you’re feeling brave, try to hack your own way to victory).

Test your hacking skills now

This Github repo - and accompanying CTF primer - is your ticket to attempting our CTF 2022 challenges, with a few notable alterations:

  • 21 total challenges - We limited our contest to 10.
  • No time limits - We had 10 minutes for easy challenges, and 20 minutes for medium ones.
  • No hints - You’re on your own.

Have fun, and happy hacking!

Sudo: a fickle beast

Our resident Site Reliability Engineer, Bob Micheletto, has a penchant for finding flaws in poorly written sudo rules. He initially wrote 21 sudo-based challenges, but to keep things simple, we pared them down to 10 for our contest. (Note: you can try all 21 here.)

Several of these challenges were derived from exploitable sudo rules I've encountered in the wild throughout my career.
Headshot of Bob Micheletto.

Bob Micheletto

Site Reliability Engineer

Network Ninja

We decided that participants in the event would:

  • SSH to a preconfigured server,
  • Examine the /etc/sudoers file, and
  • Attempt to gain root access by exploiting those sudo rules.

If and when they got root, hackers would run a special /root/ctf.sh command, which would send a secret hash and automatically update the leaderboard.

Come work with us!

We’re 100% remote and loving it. Check the jobs page for opportunities to join our diverse team.

Technology to run the event

Server setup

We considered a few hosting options:

  1. Docker containers with screen recording & manually submitting solutions, which we ruled out because they’d be too difficult to support. (Note: if you want to attempt the challenges yourself, you will be using Docker.) ❌
  2. Shared AWS ec2 instances, which were nixed because it’d be too easy to accidentally disable the system for all - plus folks might leave clues to challenges laying around. ❌
  3. Single AWS t2.nano instances for all, to be refreshed every day. It cost less than $5 for a server month, and we’d only keep them up for a couple weeks anyway. Perfect. ✅

TL;DR - folks would ssh to their own t2.nano instance each day, and figure out the challenge from there.

Spawning the servers

Each challenge server was provisioned using AWS CloudFormation and configured using our agentless Puppet configuration. Our cloud-init script fetched the Puppet configuration from code-commit and applied the appropriate challenge tailored for a specific user based on the EC2 instance tags.

User accounts & authentication

We used a Puppet module to create users on our servers, and user assignment is done through Hiera, so we generated a bunch of configuration files for the challenge servers that put only User X, on Server X. “We didn’t want anyone to be tempted to log into another user’s challenge server,” Bob explains.

We added our users public keys using the same module that adds our users. The snag was, we were going to be deleting and recreating these servers 10 times a piece. To avoid users running into SSH warnings about changing host keys, we decided to preserve the keys and provision them through Puppet.

Time limits

To enforce the time limits for challenges - with automatic logouts after a specified time - we used a shutdown trick from Unixdaemon.net. “We needed a way to keep the challenges short and sweet for our contestants,” Bob says. “Adding time limits - 10 minutes for easy challenges, and 20 minutes for medium ones - proved ideal.”

Here’s the Puppet template we used to create the timer events on login:

#!/bin/sh

if [ "$PAM_USER" = '<%= @user %>' ] ; then
    # Start the events
    echo /usr/local/bin/shutdown_timer.sh | /usr/bin/at now
    echo /usr/local/bin/hint.sh | /usr/bin/at now
fi

exit 0

Note: we still wanted the ability to login to these servers with administrative user, so we check the PAM_USER environment variable - and only kickoff the events if the challenge user logs in.

Here’s the Puppet template file we used to generate the shutdown:

#!/bin/sh

# Shutdown Delay
sleep <%= @shutdown_delay %>

cat /usr/local/shutdown_message.txt | /usr/bin/wall

/usr/sbin/shutdown -h now

Note: the duration of shutdown_delay was different depending on the difficulty of the challenge (10 minutes for easy ones, 20 minutes for medium).

Scoring

For automated scoring - with a leaderboard contestants could check at any time - we wrote a script that:

  • Sent out a statsd metric, and
  • Sent a verification email (using AWS SNS) with a secret string for verification, unique per user, per challenge. Without this, anyone could “hack” the leaderboard by spoofing statsd messages - and that’s not the type of hacking we’re looking for here.

We then built a leaderboard using AWS managed Grafana, and allowed our users to login to the leaderboard using their usual SSO logins. Here’s a glimpse of the final standings:

Screenshot of the final CTF leaderboard standings.

“Automating the scoring process took a bit of wrangling, but saved us a lot of time and manual work in the end,” Bob concludes.

Providing hints

Automated hints appeared for contestants halfway through each challenge using wall. The hint timer was set at the same time as the shutdown timer (on user login). “Frustration would ruin the fun, but we also didn’t want to give away too much,” Bob says. “Our hints provided just enough info to point folks in the right direction.”

Here’s the Puppet template we used to send the hint:

#!/bin/sh

sleep <%= @hint_delay %>

cat /usr/local/hint.txt | /usr/bin/wall

exit 0

The hint.txt file was also configured via Puppet, and was specific to each challenge.

Network Ninja branding

A special flourish for contestants was this custom Network Ninja ANSI MOTD that appeared on login.

Screenshot of Network Ninja ANSI logo.

Support, documentation, & fun provided via Slack

Each morning at 9am ET, we’d announce the daily challenge in our company Slack at #ctf2022.

Screenshot of the final CTF leaderboard standings.

Contestants had 24 hours to complete each challenge. For educational purposes, we also provided reference solutions for the prior day’s challenge in the channel.

Some challenges had multiple possible solutions, and it was cool to see people figure out their own solutions I hadn’t anticipated, too.
Headshot of Bob Micheletto.

Bob Micheletto

Site Reliability Engineer

Network Ninja

“I accidentally trashed the sudoers file”

Here are a smattering of contest highlights from Slack. Note to the reader: don’t do the following.

Screenshot of Slack message from CTF.

Hacker creativity reigned supreme.

Screenshot of Slack message from CTF.

After several lockout notices, this warning attempted to keep people from harming their own shells…

Screenshot of Slack message from CTF.

…But not everyone listened.

Screenshot of Slack message from CTF.

Seriously, it’s really easy to break things with sudo/sudoers!

Screenshot of Slack message from CTF.

Screenshot of Slack message from CTF.

When stuck, extra help was available upon request.

Screenshot of Slack message from CTF.

In contrast, others gloated about their hacking prowess, in song form.

Screenshot of Slack message from CTF.

And now, a final song to send us off. We can’t wait for #ctf2023!

Screenshot of Slack message from CTF.

CTF FTW

Throughout the competition, we had fun, collaborated together across product teams, and learned some things about security along the way. Be sure to check out Part 1 of our CTF event writeup if you haven’t already, and if you want to participate in the competition next time, come work with us.

Date

Reading Time

9 minutes

Category

Network Ninja

Are you a developer? We’re hiring! Join our team of thoughtful, talented people.