.png)
So It Begins
It was 5:45pm on a Friday. A long holiday weekend was coming up, the sprint demos were over, the wraps were complete and stories were set for after the break. This engineering team was winding down for some rest and relaxation, both few and far between for this startup. They found strong product/market fit and were starting to reach the scale of users and revenue that come with it. The sense inside the company was this thing they built might indeed be the rocket ship that is the ambition of every builder.
The alert could have been so easy to ignore.
A security information management service was showing an anomalous number of 400 and 500 errors coming from an internal tool. The company’s product was a data platform housing sensitive customer data for their users. The internal tool throwing the errors was an administration panel used to manage individual customer accounts for that product. All the errors in the panel were triggered by a single account, which belonged to an engineer on one of the product teams.
A few dozen abnormal but not immediately catastrophic errors would be attractive for any engineer to dismiss, particularly right before punching out for a long weekend.
Fortunately, someone on the team did respond and took some time to dig into the anomaly. The account in question appeared to be requesting URI paths that didn’t exist and making requests that didn’t occur naturally in the application. It didn’t take long for a clear pattern to emerge.
The logs showed it plainly - an external actor was inside the administration panel.
The gut of the responding engineer drops. The page goes out.
Someone is inside the admin panel.
And they can do whatever they want.
True Stories of Authorization Nightmares
As this technical leader of the product engineering and security teams shared his story, I reflexively groaned. Having sat in my share of war rooms following high severity incidents, I knew this one was going to be painful. This was yet another authorization nightmare.
Over the past few weeks, I spoke to software leaders about some of the worst moments in their careers. These leaders shared the high impact, company trajectory altering security events that resulted from poorly implemented, inadequately scoped or straight up missing authorization systems. These veterans of explosive growth software companies (whose brands you would immediately recognize) agreed to share their authorization nightmares with me.
They shared these stories confidentially with the hopes that the lessons could be learned without the pain they experienced. While the details are deliberately fuzzy to maintain the confidence of the companies and careers that produced them, these stories are real.
This first one in our series is the story of the Pwned Password Data Pilferer.
The Attack
“They got in from a weak password,” this technical leader shared with me whom we’ll call Bob. “It belonged to an engineer and was used with a few different accounts.”
As great services like Troy Hunt’s Have I Been Pwned illustrate, shared passwords make these authentication compromises routine. A password used on a phpBB site breached a decade ago will end up getting added in csv and txt files that are torrented and shared over and over again. Every account where that password is reused represents a new exploitation vector.
In this case, it looked like the attacker bruteforced the target account using such a list of compromised passwords.
“It wouldn’t have been that big of a deal except that when you typed our domain name and /admin , it took you to an administration panel that had no fine grained access controls,” Bob said.
“If you had access to that admin panel, you could do everything.”
The God Mode Admin Panel
A weak password is regrettably avoidable, but what turns this routine infosec event into a true nightmare was the target: the internal tool managing all the customers. Like many companies exploding with growth and barely keeping the platform operational, this startup was struggling to maintain pace with their own success. Internal tooling for hypergrowth startups are often resourced at a bare minimum to keep the engineering teams cycles invested in what will ultimately win the market in which they are competing.
This results in what I refer to as the God Mode admin panel: a hastily slapped together management app with little to no authorization or fine grained access control. All the data and functionality were secured by authentication alone. No one designs or intends for these tools to become a massive security risk; they are a function of growth.
As fine grained control becomes more imperative, so too do the demands of the business. Product features become blockers to big ticket revenue. Bottlenecks start showing up all over the stack. All the compute, storage and network bills start adding decimal places. Given how long and painful authorization implementations can be, the decision to invest there instead of one of the other thousand needs for a hyperscaling engineering team becomes difficult.
But the risk continues to grow. Every account in a God Mode admin panel can do anything.
Bob confessed, “Several hundred people had access to this panel. If one password gets popped, it can spiral out of control quickly.”
Such a nightmare compromise is what occurred here, turning a routine account administration task into a pants-on-head, klaxon inducing five alarm fire.
“The Kid in the Candy Shop”
After discovering the access, the attacker kludged through the interface by trying to guess the URI paths for customer accounts.
“What slowed them down was guessing the customer slugs,” Bob shared. “There was a way to list all the customers - fortunately for us the attacker never found it. So they just typed in guesses at customer names in by hand until they found one that worked.”
It soon became clear from the initial log review this attacker was targeting customers in the cryptocurrency space. Bob said, “Because of our one-size-fits-all admin policy, if the attacker guess the slug correctly, they had full access to the customer’s account.”
The attacker would eventually access around a dozen customer accounts. This administration panel had a lot of sensitive metadata, but what the attacker found next escalated the incident into Severity 0.
Bob’s face would turn ashen as he described how the incident developed.
“Truly a disaster scenario.”
The Ultimate Breach
With a couple of Bob’s customers identified, the attacker then stood up a free account with a data warehouse provider. Using the bulk export APIs offered by the product, the attacker exfiltrated as much customer data as they could.
“Like, all the data. All of it,” rued Bob. “We got insanely lucky that we figured it out only a couple hours after the compromise. If the attacker found the endpoint to list all of our customers, they could have exfiltrated literally everything we had.”
As a data platform, the attacker had the crown jewels. Export APIs are a common feature in the product category; a feature that God Mode access turned into a company changing event.
“It doesn’t feel that important to do the insanely painful implementation of RBAC on an admin endpoint. Until this happens,” Bob said.
“This is all we should have been doing for the previous six months.”
The Fallout
Mitigating this incident would end up taking Bob and his teams over a year to resolve fully. First, they had to reconstruct what occurred.
“Two or three days after were a painful log reconstruction exercise,” Bob shared, wincing.
Like the authorization tooling, the audit logging and security monitoring were also underinvested. Comprehensively assessing the impact involved engineers manually tunneling into virtual machines and aggregating the attacker’s path by hand.
Then there was the matter of communicating with the customers affected.
Communications around authorization nightmares carry an agony of cascading effects. Bob said, “We had an insane journey in front of us just to figure out disclosures.”
The data exfiltrated not only belonged to the customer, but to the customers of those companies as well. This meant any disclosure Bob would make would ultimately need to also be relayed to the customers’ customers as well, creating an amplifying effect to the communication complexity.
“It was a trust defining event - in an entirely negative way,” Bob said. “We had an insane amount of eng work both in terms of emergency effort exhausting several dozen people who ended up helping. But then we had multiple quarters of work implementing RBAC ourselves.”
The company’s board also got immediately involved, demanding more robust risk assessment, disclosure and scrutiny on progress. For everyone in a position of authority around Bob’s career - his CEO, his board and his customers - that Friday afternoon completely reoriented their security posture.
“We knew if we had another incident like this, it could kill the company.”
Conclusions
Eventually, Bob got through it. Brick-by-brick, the betterments got shipped, the vulnerabilities got mitigated and - eventually - the relationships with the customers were rebuilt. But to hear him talk about it years later, it sounded like one of the lowest points in Bob’s professional life.
“RBAC was an XXL project with no business value getting created. It is the definition of an all downside project. It takes forever, you can mess it up and no one is getting promoted for doing it well,” Bob intimated.
Like many engineering organizations, Bob’s looked into authorization and saw a huge timesink with little differentiation for the business coming from it. “Two things are critically underappreciated with these projects,” Bob declared. “The first is the maintenance work is where the real total cost of ownership comes from. Spiking up the project is the fun part - keeping it running and secure is where the hard part comes in.”
Bob continued, “The second is opportunity cost of time. When you’re trying to win a market, almost all your time and attention should be spent on winning that market. That’s why these projects go unfulfilled for so long.”
As we were wrapping, I thanked Bob for sharing his authorization nightmare and briefly shared a couple of my own. As we were finishing commiserating over these nadirs of being in the software business, Bob offered some advice to any who might be in a similar stage of growth.
“If you’ve never had an insanely bad security event, I would keep it that way,” sighed Bob, solemn and grave.
“You have no idea how terrible it can be.”
Check out Authorization Academy if you would like to learn more about avoiding your own authorization nightmare. If you're in the middle of evaluating your authorization options, why not book an authorization coaching session with one of our engineers?
