Tucked away in a room inside the Mandalay Bay Convention Center, the Black Hat NOC reflects a night-time view of the Las Vegas Strip.
While a screen projects visuals of old hacker movies inside the dark room, streams of blinking dashboards light up the tables in the Network Operation Centre at the US cybersecurity conference.
This is where vendors, rivals, threat hunters, and network engineers gather to build an enterprise-grade network from the ground up in less than three days. This isn’t just any network, however – it is one of the most hacked networks in the world, battle-tested live in front of 25,000 of the world’s foremost security experts.
“We replace every router, switch, firewall, and access point in Mandalay Bay,” says Neil “Grifter” Wyler, VP of defensive services at CoalFire, who runs the defence programme alongside Bart Stump, managing principal at CoalFire. “We do it so we have full control. If something goes wrong, and it will, we can’t wait for hotel IT to respond. We have to handle it immediately.”
“This isn’t a normal network,” Wyler explains. “It’s like trying to find a needle in a stack of needles. Most of the traffic would be considered malicious in any other environment, on any given day. Here, it is just Tuesday.”
Coordination
Trainers bring offensive tools, zero-days, and experimental malware as part of their labs.
One year, an instructor crashed the network mid-class for a short time by exploiting a vulnerability in the exact Cisco access points the NOC was using.
Another time, a student finished a web hacking course and immediately went to launch an attack on their local police precinct, Wyler says.
Back in 2002, when Wyler first took charge of the Black Hat network, the team supported 1,500 attendees and a few training sessions. Now, they’re powering more than 100 training classes and a crowd that rivals a stadium concert.
This needs tight coordination between all involved, even outside the walls of the conference and along the entire strip of Vegas.
“There’s an intelligence-sharing call every day with security teams from across the Las Vegas Strip, Caesars, MGM, Venetian, everyone, plus us and the police,” says Wyler.
“We talk about what we’re seeing, what threats are moving around. If something happens on our network, there’s a good chance it’s not isolated. The entire Strip is listening.”
Going hunting
To defend against malicious activity, the NOC is staffed by hunters. “We don’t rely on alerts,” Wyler says. “They’d bury us. We look for anomalies, outliers, things that don’t fit. AI helps. So does context. If 40 students in a class are all launching the same exploit, it’s probably a lab. If one person’s doing something different, we zoom in.”
That anomaly-hunting sometimes uncovers more than the team expects. In one case, they intercepted unencrypted traffic that included a man’s license plate, home address, and family photos. “We thought someone’s safety was at risk,” Wyler says.
“It turned out a private investigator was tracking him over potential corporate IP theft. That was the only time we brought in the FBI.”
Despite otherwise being competitors outside the glass walls of the NOC, inside it is all about collaboration. Andy Piazza, senior director of threat intelligence at Unit 42, Palo Alto Networks, says the mix of people and technology is what makes the NOC special.
He adds: “There’s a small team of Black Hat employees, and everyone else in the room are volunteer from partner vendors,” such as Cisco, Palo Alto, Corelight, Lumen, Arista and Netwitness.
“We share access to tools, collaborate live on integrations, and sometimes even fix things on the fly.”
“You’ll have moments where someone from Cisco is looking at their data in our tool and says, ‘Oh, that’s what my data looks like in your platform? Hold on.’ Then our engineers tweak it. We’ve even done hotfixes on-site that end up in production months later. It’s amazing.”
That cooperation mirrors what enterprise customers experience every day, they say.
“Our users don’t rely on just one vendor. They use a mix. So, we have to make sure our tools display data in a way that makes sense across platforms. That’s how we build real-world solutions.”
At the end of every Black Hat’s events – which span Vegas, London, and Singapore – the NOC team holds a public debrief. “It’s one of the most attended sessions,” says Wyler. “People want to know what worked, what broke, and what threats we saw. And we tell them.”
A visualisation of network traffic
Even the dashboards inside the NOC are open-sourced. One engineer built a visual traffic map on-site in Singapore and posted it to GitHub days later. “People love it,” Wyler says.
