To have created the position of vice-president of customer trust yourself just weeks before your organisation becomes the focus of a major cyber security incident demonstrates the kind of prescience that an end-of-the-pier fortune teller can only dream of.
But that is exactly what happened to Ben King, who after a two-year stint running regional security for authentication specialist Okta in EMEA and APAC, established a customer trust function within the business as a means of elevating the outward-facing bits of its security team.
“Since then, we have been putting more structure on the bones of how we as a team speak to customers, the market, prospects, governments or regulators,” he says. “I report to David Bradbury, the global chief security officer, and I’ve now got regional CISOs reporting to me, an assurance team who do things like respond to due diligence questionnaires from customers, and an outreach function that creates blogs and thought leadership pieces.”
Unbeknown to King, while he was setting this function up in February 2022, the incident that would make Okta front-page news in the tech industry was already unfolding via a compromise at another organisation, Sitel.
This breach was the work of Lapsus$, a cyber extortion gang that exploited failings in multi-factor authentication (MFA) to compromise its victims and hijack their data. Although it never deployed any ransomware, it hit multiple tech firms in early 2022 during a four or five-month spree and its activities continue to this day, even though a number of teenagers were arrested and charged in the UK in connection with the attacks.
Okta was caught up as an innocent bystander, and the first indication that something had gone horribly wrong came to light on 20 January 2022 at 11:20pm GMT, when its security team received an alert that a new password had been added to a Sitel employee’s Okta account from a new location.
The target did not accept an MFA challenge, which prevented access to the Okta account. The team investigated the alert and escalated it to an incident. Shortly after midnight on the morning of 21 January, the rogue account’s Okta sessions were terminated and its access suspended.
Later that day, the Okta security team shared indicators of compromise (IoCs) with Sitel, which told them that it had retained outside cyber forensic support following an incident. With the problem apparently contained, the incident was stood down pending the full investigation, which was presented to Okta in a summary report on 17 March.
However, five days later on 22 March, Lapsus$ shared screengrabs of Okta’s environment online, customers duly panicked, and the rest is history.
Doing the right thing
Looking back on the incident, King says that everyone and everything involved – including Okta’s zero-trust technology which correctly identified the initial trigger and stopped it – hit their marks pretty much perfectly from the outset.
“They [Sitel] did all the right things – access was suspended on our site and theirs, they engaged third-party forensics,” he says.
However, it took some time for the forensic report to come through and then another week after Sitel got it for Okta to get visibility of it, says King. With the benefit of hindsight, it is now clear that these gaps caused problems in terms of how Okta’s well-intentioned response ultimately came across.
This problem was compounded because Lapsus$ had posted the screenshots it had been able to grab during the brief period it had access to the Sitel workstation. It must be stressed at this point that these screenshots were obtained via the digital equivalent of shoulder-surfing on a crowded train – no systems belonging to customers of Okta were compromised, and neither was any critical data belonging to Okta or its customers exfiltrated.
Nevertheless, the release of the screenshots by Lapsus$ elevated the narrative from a minor compromise that had in reality already been effectively dealt with to a global cyber security news story. Reflecting on those days, King says Okta struggled with communications.
Communications breakdown
“We had a lot of people saying to us, ‘You came out in your blog and said you knew about this in January’ and we correlated these events and said, ‘We were aware something was going on’, but a lot of people were saying, ‘Why didn’t you tell your customers in January?’. But as far as we knew, we had a failed account takeover, but no compromise,” he says.
“While we were running a live incident, it’s very difficult for us. We can’t speculate, we need to have facts. So we had to run our incident before we could communicate. And I think that that gap when we couldn’t communicate with a lot of detail made it very difficult for us from a trust point of view.
“My manager spoke about this in terms of this being something we practise and tabletop. It was disappointing for us because we tabletop these sorts of incidents with the right people in the room, including the CEO, with comms teams, and when a real incident hit, we feel like we let ourselves down a bit in terms of the communication.”
“We were trying to let our customers know that they didn’t have to panic”
Ben King, Okta
King adds: “We were trying to let our customers know that they didn’t have to panic. I was speaking to customers who were about to reset millions of user passwords and I said there’s no scope for any of our support engineers to view passwords or reset them. If they do reset them, they get pushed through the email channel to the end-user to do that flow. The scope of compromise at Sitel wouldn’t have necessitated password resets.
“But we had some very anxious customers, obviously, and because we learned of this at the same time as the market did, we were paddling very fast under the water.”
Okta’s silence quickly drew speculation and criticism from high-profile names in the security industry, ordinary people on social media, its competitors, and even Lapsus$, which was pushing back against Okta’s response and even claimed its victim was lying. Months later, this issue persists, says King.
“We are still getting a lot of people jumping on the back of that trying to pick holes in what is a very successful, very secure service,” he says. “I’m not a media expert, but it creates clicks, it creates advertising revenue, I imagine, and if it’s a hot story just for a day, I think people see that as a win.
“But it’s very difficult when we don’t have the facts ourselves to go correct the story, and when we do and when we have, there is often not much of a story left to talk about.”
Thousands of calls
Wisely, Okta moved quickly to establish dialogue with its customers, rather than spending time ringing up journalists to ask for corrections in news articles. Within a few days, it was able to share full, unredacted incident logs with every one of the customers that were potentially impacted. It notified just over 250 customers, although only two had any of their data viewed by Lapsus$.
“We had gone out and notified a much larger number because that’s how we would have wanted to be notified if we were the customer in this instance,” says King. “If there was any risk, we’d want to know.
“We gave them full logs, we stepped through the logs, we did everything we could possibly think of to try to rebuild that trust. I was on tens, if not hundreds, of customer calls. The other leaders within security and at Okta, likewise, were on hundreds of calls. All up, we probably did thousands of customer calls, some with multiple customers in the room.
“From where we’ve landed at the end, I feel like we’ve regained that trust. In fact, I’ve had some really good feedback from many customers saying we’ve done the right things and it was obviously a difficult situation.
“A lot of CISOs who have been through an incident really felt for us in terms of the speed at which we had to react and communicate. And in many regards, I feel like we are closer and have better trust with our customers, having engaged with all of them so deeply, so recently, than we did before the incident.
“But it was very difficult in the media to respond when we didn’t have all the answers.”
Change ahead
The changes Okta has made and continues to make following its experience are twofold. Firstly, there is a renewed focus on managing third-party risk that goes much deeper than it ever did before – King himself concedes that Okta may have put more faith in third-party attestation, such as SOC II reports, than was wise.
As a result, third-party suppliers like Sitel (with which Okta no longer works) can no longer mark their own homework in this way, but must be subjected to a much deeper level of scrutiny, more akin to something a bank might demand. King, who before moving supply-side, ran the international, then European cyber function at Australia’s Commonwealth Bank, has a good deal of experience running this kind of compliance regime.
“One of the first things we changed immediately was auditing our material suppliers in that regard, regardless of their SOC 2 compliance or ISO compliance or whatever else they might have to show us,” he says.
When it does choose to work with a third party, Okta will also no longer allow it to use its own equipment to access Okta’s systems – the Lapsus$ breach having occurred entirely within a Sitel workstation.
“Going forward, any customer support engineers or firm that is doing a similar service for us is mandated to use Okta endpoints,” says King. “In this incident, we were somewhat hamstrung in our ability to respond and communicate because we couldn’t investigate what had happened. We had to rely on the third party and their forensics team to tell us what happened.
“But going forward, anyone supporting Okta has to use our devices so that we can make sure it’s patched, make sure it’s in good health, make sure we can monitor it. And if there was another incident like this, we would be able to investigate and remediate much more quickly.
“It might mean we can’t look at some suppliers going forward, or things are a bit more expensive as we supply laptops to third parties. But that’s the cost of security.”
Secondly, in terms of how Okta goes about communicating security alerts or incidents to its customers, King is making changes based on feedback gathered from its customers, many of whom said the disclosure process was not as smooth as it could have been.
As a result, Okta is taking a number of actions. Firstly, it is establishing a dedicated security contact at every customer to serve as a dedicated touchpoint for Okta’s security team to talk to. This does not have to be one individual; it could be a mailbox that every member of a customer SOC team has access to.
Then, later in the year, it will create a dedicated security channel to share information and data with customers. The details of this are yet to be finalised, but King is adamant that both channels of communication are kept strictly on topic. “I’m a security professional too – I get contacted more than I want to be by vendors,” he says. “I love to over-communicate, I think it’s great, but it has to be legitimate communication.”
Learning experience
Ultimately, nobody in any organisation – let alone one devoted heart and soul to cyber security – sets out to be attacked, and being subjected to a supply chain incident over which you have even less control than you would if the attacker breached your own systems is a uniquely painful experience.
But it is also a learning experience, if you are prepared to draw lessons from it. Indeed, a recently published UK government study found that in many cases, the experience of a cyber incident was worth a lot in terms of getting company leadership to take notice of security issues, and there is even evidence that experiencing a breach can be good for a security professional’s career.
In Okta’s case, its experience, although thankfully mild, has galvanised policy change and forced a new awareness of how incidents are perceived beyond the bounds of the organisation that will, hopefully, be of great use going forward. As King observes: “The benefit of an incident in security teams is worth a lot.”