INL cyber training

I had the chance last week to attend a 4 day training course with Idaho National Labs in Idaho Falls. The class was 'Detect the Attacker (311)'.

Note: This blog ended up a lot more technical than usual and is a sold wall-o-text.
You have been warned. (I'm not even going to pretend I'm sorry or explain all the tech stuff. If you have questions, you know how to reach me - happy to explain it).

I want to start by saying that, as most long term readers know, I used to teach all things Opto about once every 4-6 weeks to around 12 students for over 15 years, so I know what its like to train and be trained. It is no small thing then when I say that this was, by a HUGE margin, THE BEST training I have been part of. Ever.

I had quite the 'fun' drive getting down there, some early winter snow sure made the drive a lot longer and slower than expected, but I was on my own, had some good music, plenty of car snacks and was not in a hurry. The wardriving rigs held up quite well. Enjoy this single image of this whole blog. Also use this break to grab a tall beverage of choice and settle in.. Its going to be a long read if you are game.

The instructors were beyond amazing (more on them shortly), and the setup was just simply mind blowing. The classroom AND the actual labs were astonishing. The neatness / cleanliness, the detail, the realism... Every single part of that building was just top notch.

Personally, I found it very difficult and really depressing to go through the process.
The depression hit about 15 minutes into the course starting.
The instructors gave some overview of the course, building layout, restrooms and drink locations etc.
They clearly made the point that NO photography was allowed. Hence this blog being pure text. Of the lab or the lab work book... So far so good.
Then they asked each of the 24 students to stand up and state their name, company and job title.
Very short, sharp and to the point intro from each student.
I was about the 14th or so, and as I listened to each one before me, my heart sank.
Everyone, every single person had ONE single focused role in cybersecurity. Be it red team (attack), blue team (defend), purple team (both), CTI (Cyber Threat Intelligence), SOC (Security Operations Center) and so on.
Why did I find that depressing? Because I have to do every single one of their roles. I am one person doing the cyber roles of a room full of people, PLUS all the other industrial automation stuff I get called on to do.
I am already overwhelmed with the cyber aspect of my job, that's why I signed up for the training in the first place and to hear that each of these guys was there to get help in their ONE role and I had to learn the lot was just sole and spirit crushing.
Second, I was seeing/ hearing that I was the oldest student there by a country mile.

At the first break, I actually went up to one of the instructors and asked if there going to be any issues if I walked out now.
Cue the amazing lecturer. He sat down with me and stared asking really pointed open questions about what was on my mind.
I ran him through why I was convinced that I was in WAAAAAY over my head and answered his questions about my role and daily tasks. He then said the bit that I needed, the life line.. Words to the effect of "Yes, no doubt, you are stretched thin and over tasked, but consider staying to learn some new techniques and tools to help offload some of the work".
That was the light at the end of the tunnel I needed. I was mentally shattered and in darkness, but thinking that I might pick up something that could help me fight my cyber battles really gave me that glimmer of hope. A real actionable reason to stick it out.
Personally, I found that advice even more impactful a week later.
The instructor showed wisdom, compassion and encouragement in his advice. It was not the usual "do this" that we often all jump to when trying to help others, but it made me pause and think about how we teach our kids and grandkids. Not just 'teach them', but teach them how to learn and grow through a difficult lesson.

So, I stuck out the 4 day course.
One of the things I found really interesting was the 'roll play' that the whole 4 days tied together. The scenario was that our factory had merged with another and at first things went well, but for some reason the merger fell through and the other factory workers were not happy. They had full access for many weeks/months and so who knows what they put in place. So it was a mix of insider threat, logic bombs and outside attack.

We started out learning the lab OT equipment (PLCs etc), learning the layout of the mini SOC (Security Operations Center) that we could model our SOC on back at our work places.
Then we jumped into OT protocols... Modbus TCP, DNP3, Profinet and s7comm.
CIT (cyber threat intelligence) - understanding threat informed decision making.
Detection Coverage Analysis. Really learning to think about how to detect a cyber attack.

SIEM - Security Information and Event Management - got lightly touched on. SIEM is about using tools to bring a lot of different sources of information together.

CTI - Cyber Threat Intelligence. This one struggle with a fair bit. Its about how you keep up with threats and attacks in real time, in other words, what email lists or websites can we use to keep our finger on the pulse of the cyber incidents that are happening each day. Tip, its a massive fire-hose and I drown every single morning in email newsletters and website alerts.

Hypothesis planing. This was a big one. Don't just guess. Don't just spray and pray with our time and defenses, but really get facts about the attack and build a hypothesis on who the attacker might be based on TTP (Tactics, Techniques and Procedures) that we are seeing used in the attack.

IDS. Intrusion Detection Systems. This was a bit one for me. They introduced me to Malcomand Security Onion two very interesting IDS packages that I plan to explore a great deal as I am VERY unhappy with our current IDS (Dragos). Since I am a one man band, I really need help getting some tools to do as much heavy lifting as I can (Mostly asset inventory (knowing every single IP address on my OT networks, what version of software and firmware they use and what is communicating with them and what they are communicating with) and anomaly alerting after base-lining normal.

Next up was Correlation. This is a critical bit that I have been missing somewhat due to feeling I had to use Dragos... But Correlation is all about joining the dots. The attack does not start on a specific PLC in the middle of the plant. The attacker had to get there somehow, so how, what dots got joined to get there. Correlate all them and find the breadcrumbs. Solid chunk of learning that I need to action more as I get Malcolm deployed. Quick example, Dragos can not ingest Windows syslog or firewall logs, Malcolm can.

Traffic Analysis class time was about reading Wireshark pcaps. Wireshark is software that can listen and log Ethernet traffic, a pcap is a packet capture of that data over the wire and learning how to deeply inspect it and understand what you are looking at - this ties in with the class time we spent on looking and learning about the core OT protocols.

Finally then the last day was spent putting all this together by actually attacking and defending a power and light control system and a very realistic mock up of a chemical treatment plant.

Two comments about the CTF (Capture The Flag - the attack and defend) exercise.
We were asked by the instructors not to use outside resources to try and keep it real and to push home the skills we learned the past few days..... Every student I saw in my area used AI on their phones every time something changed in the lab or when they got 'stuck'.
Second to this, all the students that were in my part of the lab could only deeply focus for about 15 to 20 seconds at a time, then they would look up, look around, tried to talk to someone else or used their phone or AI, then they would re-focus back on the task on their computer for another 15 or so seconds. It was astonishing.

Another thing I found interesting was a good number of younger students who mumbled. They simply seemed unable to speak up or speak clearly at any point in the 4 days. How they got through the job interview process or how they were the pick of the job applicants beats me.

So I finished the course, I got a good insight to some new tools, I learned new ways of looking at things, I got a 2. 5inch thick manual full of my notes of stuff I need learn/re-learn. I got overwhelmed. I got stuff to do. I just gotta get up each morning, show up, don't complain and get it done. Leave the place in better shape than I found it.

At the end of course we were all handed a really nice weighty challenge coin as the only physical proof we were there.

Ok, I'm at 1,613 words, this has run long.