FireflyHacker's CCDC Experience

I love the competition and I am a firm believer in the open sharing of knowledge and education.

My philosophy was that if you throw 30 students in the deep end and 8 of them figure out how to swim you will have a good team ready to go. The time frame is extremely tight to learn an insane amount of new technologies, strategies and the “meta” of the competition. This is (roughly) how our weeks we structured throughout the year.

UCI’s 23-24 CCDC Strategy:

Background: During the 22-23 season the UCI CCDC team did not do well. As a team we were a mess. We were unorganized, relied on strategies that weren’t changed or modified in a few years, and every leader on the team was just not well enough prepared for being a leader. The consequences of all of this was a team that didn’t even come close to qualifying, and a bunch of members who were ready to call it quits on the entire team.

The new(ish) hardware and room: We were lucky to get a room for the club. I am still unsure of the exact details of how this happened, but I know that by the time the qualifiers were over we had our own space with a server rack and 3 servers. The servers were running esxi and we had managed to get 1 practice network out of them. The issue was the licensing was set to expire in less than a year and we could not get any confirmation that we would be getting updated license keys. This was where I decided (after a lengthy conversation with Charles, and Cara) that we would be moving to Proxmox to avoid license issues moving forward. The room at this point was empty. There were about 30 chairs and 8 tables. No cables, no monitors, no workstations, no printer, no phones, nothing. It was an empty room with a server rack. We knew there was tons of potential we just needed help acquiring the hardware to make the room a real hackerlab.

My general plan for the next year: I refused to give up and I insisted on starting CCDC practices early. We had an entire spring quarter (10 weeks) of school left and I wanted people to get started on CCDC as soon as possible. We were starting a fresh team with limited experience, I wanted people to come in before summer, get some practice in during summer, and then be ahead of the game by fall when we would get the biggest influx of new members. So I started announcing CCDC meetups and holding them in the new room we had. They were very informal and mostly just seeing who was showing up and who wanted to be a part of the team. During this time it quickly became obvious who was actually left from the previous year’s team that cared to move forward. Two new members, Charles, and Cara, were still helping decide meeting times and attending practices. I wanted to take over the team but knowing my own failures as a leader of the windows team I wanted to run the team with another person and Charles stepped up to that role. The three of us then started building things out for the next year.

Building back up over the summer: Charles took my vision of having training over the summer and made an entire training program. He broke them down into blocks and released each block on the discord server over the summer. Unfortunately not everyone is as dedicated and as the blocks were released fewer and fewer people were doing them. During this time we had done a fresh install of OpenStack on the 3 servers we had, and had gotten 2 r620s that were running proxmox. We found OpenStack difficult to use and at the time I found it hard to understand how cloud networking works, so we were almost exclusively using the two proxmox machines.

The New School Year (23-24):

After the qualifiers I was still interested in getting our new servers to a point of being usable. We had 3 dell poweredge servers and they were all running esxi, the goal was to get them to a point where it was easy to use them for setting up practice networks. I wanted them to be setup and ready to go for the next year so that I could implement my strategy of “Practice how you compete”. In other words, I felt that the best way to train people is to put them in a practice that is very close to the competition. I fell in love with CCDC because I loved the chaos of getting into a comp and everyone is yelling instructions at each other and things are going down and no one knows what the fuck is happening. It moves quick, it is chaotic, and when you have a team of people who all enjoy each other’s company there it makes it really fun. I wanted to suck in new members by letting them experience that early on in the team practices.

Week 1 (Planned): Intro to CCDC, what is it? Why should you join? What is UCI’s history with the comp? What is Red team, White team, Orange team, Blue team, Black team? How are the points broken down? What are the limitations/rules? What is the main objective? Then if you want people to join tell them how it help them get experience with many types of technologies and all the resume potential.

Week 2 (Planned): Explain each subteam if that is something you want to do. We had a Linux, Windows, and Corpo team, with a Captain (Charles) and a networking person (me). If you want to go the route of sub teams you should explain the responsibilities of each team. Then you should go over the basics of each team with everyone (ex: corpo: what an inject looks like, linux: basic command line navigation, windows: what windows server looks like and how to use it.

Week 3 (Planned): This was where I made everyone start building the first practice network. We guided them through using virtualbox (makes exporting to proxmox easier) to make a vm depending on what team they were in. Everyone got general guidelines for what we wanted to see, and some people had to work together to build services that were dependent on each other. We encouraged people to use really old versions of windows/linux and to stop updates from happening while they were setting it up. Also to look up best security practices and doing the opposite.

Week 4 (Planned): Helping people finish building their machines and importing everything into the comp environment for the practice on the weekend. The weekend practices were 100% replications of what we expected to see. We started at 9am and went until 4 or 5pm. We used a scoreboard and had our corporate lead make injects to do throughout the comp. We also always made sure to have a red team. I recommend trying to find other college teams that practicing for offensive security competitions like CPTC or CTFs and see if they would like the practice against an active blue team. I also recommend finding people to act as orange team and call throughout the day with questions and problems that the team has to solve while handling the red team and the injects.

Week 5 and Week 6 (Unplanned): Go over the issues discovered during the practice. What went well with the practice? What went poorly? How did people feel about it? Was it harder or easier than expected? What was the score breakdown? Go over the injects and explain what was good or bad about each one. Later in the week start getting into sub-team specific practice like linux and windows functions and administration.

Week 7 (Unplanned): We had a invitational around this time. We ran it as another practice for new members keeping an eye on how each person did and evaluating what the skill gaps were. Then we worked on improving the things that people were unsure about. At this point members that are not putting in any time outside meetings are going to be falling way behind and will likely not make the team. Encourage everyone to get some practice outside of the meetings and offer projects that they can work on.

Week 8 and Week 9 (Unplanned): Prepared for another mock comp on the weekend that we imported from WRCCDC (this is important so you get practice in an environment that you are not use to seeing). We had another school (CSUF’s CPTC team, Thank You!) help us by being our red team, I took the role of the Black team and Orange team, Cara took the role of White team, and Charles was getting practice as team captain and also Black team consultations.

Week 10 (Planned): This is around the time we announced our team. I always advocate for offering people the opportunity to find out why they were not selected for the team, it is a learning experience and can provide achievable goals for the next year and an amazing learning experience, but I don’t think anyone took us up on our offer.

Competing with the new Team:

Invitationals 2: Once we had our team selected we competed in invitationals #2. We got the opportunity to see how well our main team did and how we compared to the other schools. This was a much needed competition where we were able to see what each member needed to work on improving. Each person had their own short comings, from the top down and everyone was clear of this. There was an open discussion where no one was free from blame and everyone had the opportunity to point out what issues arose. You have to have good repore with your team, this would not have worked if everyone was scared to speak up, or if members refused to take responsibility for their shortcomings. This was also useful to see what strategies were working well and what still needed improvement, and the same goes for the scripts.

Qualifiers: After the invitationals we did a few more half-mock comps with a full network but no red, orange, white, or black team. We used these competitions to hone in certain skills we had to build. For me it was firewall practice with VyOS, OPNSense, Cisco ASA and FTD, and Palo Alto, for linux it was scripting, k8s, rancher, and AWS. We also were working on refining our scripts and I made my favorite program yet (Passgen write up coming soon).

Regionals: After qualifiers things got real. Our main problem was that with such a new I was the only member who knew what it was like to compete in person. This was the timeframe where we practiced getting use to deploying and using our discord replacement as well as testing our scripts. We decided to do one big mock competition a week before regionals that was an exact replica of the previous years regional competition that we failed to make it to. We deployed the 2023 WRCCDC regionals network exactly how the network map showed. We secured 10 laptops from NUCC and someone we met at the local meetups IVU and OC2600 and installed a mix of windows and linux including a few OS versions that we didn’t expect to work well (XP, windows 8.1 and TailsOS). We had 2 workstations that were running Proxmox with the “on-prem” parts of the network. I also went out and bought the exact camera model that used in the previous year’s competition so that we could practice patching and securing it. I found an old access point I had to use for the WiFi and we asked around the community for a Cisco ASA and catalyst switch since we were told that we would have ASAs and Cisco switches. I used the free Palo Alto VM from the Palo Alto training to combine the “hybrid” and on prem networks to our larger comp net with a 1:1 NAT matching the WRCCDC competition almost perfectly. I also called in a few favors and got help from a few people who were able to act as Black team and Orange team. I also set up a VOIP phone with FreePBX and SIPstation so that our Orange team members could call us and ask for help just like in the real competition. On top of all of this I had someone come in to the lab where we were practicing to act as a customer in person. We also made injects and a team packet for the practice. Thanks to Charles we were able to get all of this deployed and ready without anyone else on the team having any idea what they were walking into. When the day started everyone walked into the room and ran through our strategy guide as if we were doing it for real. I had to play dumb on a few things since I had insider knowledge from setting things up and I had to clarify a few of the injects, but otherwise it went incredibly well. We identified a number of issues with our strategy during the practice and found that we were slow to get certain tasks done. I firmly believe that without this practice we would have not done as well at regionals.

WildCard: We took 2nd at regionals. The bad news was that we lost to the very talented people over at Cal Poly Pomona. The good news was that we weren’t out of it yet we got another shot at making it to nationals. More bad news the wildcard was like 1 weeks after regionals and it is based off a completely different type of competition called CyberPatriot where the goal was to remove malware, fix GPO configs, update software and answer forensic questions. We were given more time to refine our scripts so we patched a few bugs we found at regionals and added a few more scripts that might come in handy for the wildcard or nationals. One of our new members Akshay had extensive experience with CyberPatriot style competitions and our alternate Athena did as well, so we handed the team to Akshay for the wildcard competition making him team lead. There was a lot of work out in to prepare for the wildcard, I remember feeling completely burnt out between Regionals and wildcard especially after learning that other teams had significantly more preparation time between their regionals and wildcard.

We won the wildcard. I would like to say it was close, but I feel that we held a pretty good lead from early on. Unfortunately I don’t have much advice for wildcard other than to point out it is a different competition from CCDC. CCDC is a service competition, CyberPatriot is a forensics competition and the overlap between a good CCDC team and a good CyberPatriot team is a lot less than you might initially think. When preparing for wildcard dig up anything you can from CyberPatriot, practice VMs, example forensic questions, learn everything about the “meta” meaning learn the patterns that frequently appear in these competitions and the starts that consistently do well.

Nationals: Nationals were crazy. We were flown out to San Antonio, booked nice hotel rooms, and once again the game changed. For regionals were were guaranteed a whiteboard, but it was 50/50 that we would get that at Nationals, so any part of our strategy that required a whiteboard had to change. Nationals also doesn’t use AWS, so those skills became unnecessary. The other big change is that the Red Team at Nationals is significantly better than regionals Red Team, plus we had 3 dedicated professional red teamers going against us at Nationals whereas regionals pools their red team against all teams. We are nerds so after we landed everyone spent a few minutes settling before we all met in one of the rooms had food delivered and continued strategizing for the competition.

The next day we all met in a large room for breakfast and some opening remarks. Then we walked to where our room was and waited nervously to start. Right after walking in we did a quick inventory of things like the number of Laptops, what OS each one was running, I started tracing Ethernet cables to make sure I knew how all the devices were connected and checked for any hidden/unknown devices connected to the main switch. On top of the switch was a brand new Palo Alto 3260 that had just been set down as we walked in the room. We decided to leave the firewall for later so that we could focus on other things at the start. Another change was that we had a hybrid network. We were given access to an ESXi server for our remote network and we had an intel nuc for our ESXi server locally. The other change was that we had Sponsors staying in the room with us as our room judge while we competed. Not a huge change and it was actually really cool that we had someone there that we could chat with when there was down time.

My job as the Network/Firewall guy:

On our remote network we were given a Cisco FTD for our firewall. The nice thing was that it was just boot looping any time I tried to use it, so I ignored that garbage and worked on the Palo Alto instead. The PA had a valid license so I enabled all the machine learning and AI features and created a new policy that would block anything marked as a medium threat or higher. I was also watching our logs for threats that were marked lower and manually adding them to the policy as needed. Mostly through the first day after I had the Palo Alto deployed and working I started seeing Silver beacons popping up as “Informational” in the logs. Well I don’t want all my Windows machines to have active beacons on them so I added it to the rule set and set it to block. I was later told that this confused the hell out of the Red Team because all the sudden they lost all their beacons but still had access via other means.

Day 2: We found that after I made a new rule we started having phone issues. It turned out that simply allowing all for the IP for the phone was not enough. I had to also add a specific rule for ntp, rtc, and rtcp. I found that out by following this guide from Palo-Alto. I tried to get OPNSense running on the remote network, but our host based firewalls were performing well enough that we didn’t see the need to do so. The few tasks that I remember attempting did not go well for me. There was one for hosting a web based file share that used permissions from the AD. In the hour we had to get it working I put together a file share using IIS and set the folder permissions to what was asked of us. As far as I could tell this was working when I tried it, but apparently the users could not log in. If I were to do it now, I might have tried Seafile, or just used an SMB share so that I could have got some of the points for that inject. I did assist with 2 of the injects because they had to do with recent vulnerabilities for the Palo-Alto VPN and Cisco ASA. I knew the basic details off the top of my head and was able to quickly pull up a couple links on them.

Knowing what I know now I would probably have changed how we structured the team. I feel that my job was just not needed overall. I’m not saying that I didn’t add any value to the team, but if I had put my focus on some of the oddball tasks that we got I think I could have contributed more. In the time since the competition I have learned a lot about SSO, file shares and things of that nature that would make me a much better teammate if I was able to compete again.

So what did I do?: My biggest contributions were more abstract in nature. I created a few strategies that I feel were very beneficial and pushed for constant improvement of our strategies. My most successful contribution was Passgen. The strategy behind this software is (in my opinion) genius. One of the meta things to know about ccdc is that you can bring paper into the competition that doesn’t have to be made public before hand. When I first joined the team we had these sheets of passwords that we would bring with us for quick reference when communicating password changes. When I wrote Passgen I designed it to work with a seed so that we could memorize a few seeds and bring in our password sheets. Then once we were in the room we could run Passgen, put in the seed that corresponds to our password sheet and get a csv with the same passwords. This let us communicate passwords quickly and take advantage of copy and pasting passwords without having to type them. The important thing about Passgen is that it is cryptographically secure and looks like a normal password generator. I made sure to use ChaCha20 and Sha256 for the seeds and value generation. You can also generate random passwords without using a seed which is useful for then taking as input for scripts. We used this function to randomize AD user passwords since we didn’t have to have a paper reference for them.

What Should Your Team Do?

I can’t tell you that. The simple fact is that every team is different. Even if you were leading the same team for multiple years in a row the training is going to change every year. I do have some advice that might be helpful:

  • Start fresh EVERY YEAR. No one is on the team by default. Everyone has to prove themselves again. EVERY YEAR.

    • The unfortunate reality is that people’s priorities change, and if someone stops showing up they should not expect to make the team. We called this nepotism, and the best teams will be the ones formed without any bias towards former members.

  • Find a way to introduce brand new members to core concepts and ideas while reinforcing foundational knowledge for more advanced members. This is fucking hard to do, but if you find a good course or build a good plan you will be at a great starting place.

  • Practice how you compete:

    • Are you allowed to use discord during the comp? How about google drive? O365? Will you have a whiteboard? Are you remote or in person? Will you have a flash drive to install a fresh OS on a laptop if you have to? Will you have licenses to use word or will you have to use LibreOffice? Find what the limitations are and plan around them. If you need something like discord to communicate across the team look into self-hosted alternatives that you can spin up quickly or with a docker container (assuming your comp allows it).

    • Sports teams practice with scrimmages and drills. You have to do the same thing. There is nothing worse than walking into a competition for the first time having no idea what to expect. It will take time for your team to adjust, so make them adjust during practices instead of waiting until you are in the competition environment.

  • Learn who your competition is. What are they doing differently? How much are they practicing? What content are they covering? Are they writing scripts? Can the scripts they write help you and your team? Do their scripts have intentional bugs in them like a wininit command that has 1000 spaces before it so you cant see it when looking at the code?

  • Learn from your mistakes. Every time you have mock competitions you should immediately follow it with a debrief and then a more detailed debrief a few days later. How do people feel they did? What mistakes were made? What went well? What tools or software could have made things better? What issues were caused by communication issues? What issues were solved with good communication?

  • “Scope out” A phrase stolen from Charles:

    • Think about the bigger picture. Your website isn’t working. Is it using a database on a different machine? How about external factors like DNS? Can you reach it by the IP address? Is your website using keycloak for authentication? Is keycloak working? Is AD working? Did something change on them recently?

    • You really want to be asking yourself and your team a few questions:

      • Was there a recent change or script run?

      • Are any other services down?

      • What are the dependencies of your service?

      • Is your service consistently down or is it failing every few checks?

    • You cannot get stuck focusing to hard on the systems that you are responsible for, you have to consider that other people are working with you and may have made changes that affect you.

    • Personal Examples:

      • I changed all the passwords on the AD. I forgot to mention this. Everything using Keycloak went down. The reason was because the Administrator password was changed without it being changed on Keycloak, so Keycloak could not authenticate against the AD. We thought it was something wrong with Keycloak itself, and went down an unrelated rabbit hole for way to long.

      • I had an issue where every few checks my service went down. I could not find the cause of it until I realized that it was using AD for auth, and that the scoring engine was rotating through different users. After checking the logs we found the user that had the failed login. It turned out that the scoring engine did not like one of the characters in the password, so we changed the password and submitted the new version.

      • Services such as DNS, AD, Databases, and Authentication can have wide spread side effects when changes are made or they are compromised. When changes are made to these services the person making the change should make it clear that it could cause problems. If nothing happens within a few minutes everything is fine, otherwise the changes need to be reverted immediately.

      • Routers and Firewalls are useful tools, but can also cause disastrous and unrecoverable problems if you make a mistake. Things like adding a firewall rule that blocks a bunch of things, or removing a rule that is allowing your NAT to work properly will bring down your entire network.

  • Create a detailed and flexible team strategy. Everyone should have a job at the start of the competition. Compromises will have to be made, if things are going well early on you may be able to get away with the big changes sooner rather than later, or if things are going poorly wait until later to make the big breaking changes. Play to the strengths of your team members. Don’t force people into roles that they aren’t going to fit in. Make the role fit them. Give them responsibilities that they are best at. Don’t be afraid to to tell your corpo person to do AWS if they have outside experience with it. Don’t be afraid to make one of your Windows or Linux people answer the phone for a customer if they are the right person for the job. If you have someone on the team who wants to learn networking well enough to take the role then let them, but if you don’t have anyone interested have someone learn the bare minimum basics and work a strategy that doesn’t require a network person.

Other advice

CCDC is a Service Based Competition: It is easy to look at CCDC as a defense or system administration competition, but in reality it is a business simulation of a IT consulting company, in other terms it is a service based competition. 60% of your points (20% customer service, 40% injects) are based on how well you treat your clients. The other 40% are for how good of a job you do keeping your clients services running. You do not get points for making your network more secure, you just don’t lose as many points. This was a major perspective shift that we had to make, and if I was to it again I would have put an even bigger focus on the “Customers”. You should NOT make changes that cause your clients downtime. If your client needs you to do something like provide a VPN for their work from home employees, then it is your job to do so and explain to them how they can use it. If they ask you for the pros and cons of the VPN you chose then it is your job to kindly explain it to them. You will lose (get fired) if you do not care about your customers. You were brought in to do a job for your clients and you should remember that. It is easy to lose the “role-play” aspect of CCDC, and CPTC does a much better job in keeping you in character, but I think it is a helpful perspective to take.