Att&cking The Engenuity Evals (Mitre by Mitre)

If the title of the article didn’t strike you as Engenuis, that’s ok, we’ll just move along.

By now, most people working in some sort of Cyber Security role would have seen at least one form of Vendor self-praise around the latest Mitre Engenuity Att&ck Evaluations for Enterprise – Wizard Spider & Sandworm 2022.

I even made a Meme about it which, if I have to say so myself, is doing pretty well on the socials.

Now, don’t get me wrong, I love vendors. I also love branded hoodies and socks and stickers and free stress balls and iPads.

Swag Sidenote

If there is a gap in the market, I’m willing to do a Vendor Swag comparison test. We’ll call it the Swanepoel Vendor SW&G Off for Enterprise. What is great about this is that there is no fee for vendors to participate, but Swag will unfortunately not be returned after the evaluations have been completed. Similar to the Mitre Engenuity Att&ck Evaluations for Enterprise, the Swanepoel Vendor SW&G Off for Enterprise will pit the best Vendors against each other, and see who has the best SW&G, without us telling you who came out on top. We do realise the first question you’ll have when hearing about the SW&G Evaluations for the first time is “who won”, and we understand why you ask. Unfortunately, while evaluations data should be informative and aid in decision making for SW&G related stuff, it becomes difficult to control how each vendor would interpret their results.

Introducing the Average Analyst

So usually when a new Mitre EDR / XDR evaluation rolls by, I try to make a Meme then don’t do much further.

But, in this the year of our Lord 2022, I thought it was time to stand up for the Average Analyst, pick up my keyboard of destiny and do the work that no average analyst has the strength left to do after a day of battling “is this the Russians or just IT doing stupid stuff” alerts. That is, to try and make sense of the 2022 Mitre Engenuity Att&ck Evaluations for Enterprise – Wizard Spider & Sandworm.

Because if we are to leave it to the 30 participating vendors to interpret the results for us, we’ll walk away thinking everyone is a winner, everyone prevented all badness and you should defo be ripping out your current solution and roll their product instead.

Don’t get me wrong, EDR / XDR vendors are special vendors. These are the folks helping thousands of SOC Analysts, Incident Response Analysts, Security Operation Managers and CISO’s sleep at night. Knowing your environment is protected by decent tech, provided by vendors who have hopefully employed some of the smartest minds in the industry.

The Mouthpiece Of The Average Analyst

So I hear you at the back. “What gives this guy the right to speak for the Average Analyst?”. Well, allow me two full sentences of self-promotion:

Sentence 1: I’ve done DFIR (Digital Forensics and Incident Response) work since before CrowdStrike was founded. (That definitely sounds so authoritative, doesn’t it?)
Sentence 2: No one else is currently speaking up for the Average Analyst, so I might not be your favourite child, but I’m the only one you have.

The point I’m trying to get to here is that if I struggle to make sense of the Evaluations, so might a lot of the other Average Analysts out there. If you are rolling your own Sysmon Elascticsearch stack on top of Kibana authentication logs on a distributed ledger to verify the authenticity of alerts, then you probably aren’t the target market here. (But still always welcome to stick around for the jokes).

Why So Confused?

If you don’t know why there is confusion about these results, it boils down to this: The way the Evaluations work means that Mitre doesn’t define a sole “winner” at the end. Thus leaving the door open for each Vendor to interpret the results in a way that they feel fit.

Don’t believe me? Let’s take a five minute Google and see what some of the Vendors say about this latest set of Evaluations:

  • Cynet ranked all 30 vendors crowning SentinelOne as the winner.
  • Cybereason said they won: “Undefeated in MITRE ATT&CK Evaluations” and “leads the industry in the MITRE ATT&CK Enterprise Evaluation 2022”
  • SentinelOne agreed with Cynet, also claiming first spot, with Microsoft second and CrowdStrike third.
  • CrowdStrike said that it’s actually them who won (leading is winning right?): “CrowdStrike leads the latest MITRE ATT&CK Evaluations”.
  • Microsoft went the humble route by not actually ranking themselves, They did say that they “successfully detected and prevented malicious activity at every major attack stage”.
  • Palo Alto in turn said they also won, going for “Cortex XDR Triumphs in 2022 MITRE ATT&CK Evaluations”.
  • Trend Micro gave themselves a Top 3 finish, while Malwarebytes also chose not to rank themselves, only saying they scored “High Marks”.
  • Finally, we end with BlackBerry just saying they were 100% successful in preventing the attack emulations.

So, Is It Really Worth It?

I don’t know, but that is what we are here to find out.

The fact is, Mitre believes so and so do 30 EDR / XDR vendors. You can rest assured that these latest Evaluations have also found their way to your CISO’s inbox. The person in charge of procurement will most likely use this to guide the shortlist of Vendors to review when renewal time comes around.

So instead of just making it off as marketing dribble, or responding with a blank face when your CISO asks you “So, based on the Evals, which XDR should we buy”, join me on a fact-finding journey and let us see what it’s all about (Ok, that sounds extremely cheesy but I’ve run out of smart things to say now).

Finally, drop some comments below to help me understand where you are at.

  • Is there anything you love about the Evals?
  • What do you hate?
  • What doesn’t make sense?
  • What is your number one burning question?
  • Do you think there is any value in it?
  • Can we all blame the poor marketing teams?

If you are a vendor and would like to send me Swag to help alleviate the pain about to rain down on your marketing team from my keyboard of destiny.. ok, way over the top, OVERRULED.

Based on the feedback, we’ll formulate a strategy and unpack the Evals in the upcoming posts. (If this article dies a slow and lonely death, humiliated by its view count, this will be the only post in the series, and also be the shocking end of me being the Mouthpiece of the Average Analyst.)