20 years of SIEM: I’m celebrating my dubious anniversary

20 years SIEM?

On January 20, 2002Exactly 20 years ago I joined a “SIM” provider that should remain nameless, but it is easy to find out. That windy winter day in northern New Jersey definitely gave my security career a new direction.

With this post I wanted to briefly reflect on that threatening Anniversary. Where do we start?

Let’s start with a sad Fact that some of the problems that plagued SIM/SEM in the late 1990s and early 2000s still exist today in 2022. Of course, one of the most notorious and painful problems that persists for an amazingly long time is that of data collection.

I remember our engineers struggling with some API based collections in 2002 a well-known firewall provider. API-based log collection seemed new and strange back then (“Why can’t they run UDP 514 syslog like normal people?”). Today, the current generation of engineers are still struggling with some cloud-based telemetry collection mechanisms… and that’s before security observability really arrives.

Another problem, as we now know, has amazing persistence. The promise “we collect data and then you get the insights” was made by SIEM providers as early as 1999.

Do SIEM operators today feel they got what was promised? Generally no. In some areas we are worse off as users have to work more, not less, to harness the value of the data they collect. Personally, I still hear SIEM users loudly complaining that “it just collects data” and “it’s difficult to get the results you want”. This is after a quarter of a century of promising that! Note to enthusiastic safety data Lakers here: Sorry guys, but you’re making this worse, not better…

Next, another reminder is useful: definitely the birth of Security Information Management and Security Event Management older the rise of compliance. In recent years, it has always annoyed me when SIEM was equated with compliance because I lived through the era before compliance requirements in security became “cool”. Thinking back to 2002, SOX was just coming out, HIPAA was new and cool, while PCI DSS… wasn’t born yet.

If you’re curious, what were people interested in back then? Here are some juicy quotes from the SIM/SEM/NSM/ITSM/LEM marketing of the era. Btw, back then many names were used for this area of ​​technology, eventually SIM/SEM won and then got merged by Gartner forever.

(Date: 2002, source)

(Date: 2002, source)

(Date: 2002, source)

(Date: 2002, source)

(Date: 2002, source)

(Data: 2003, source)

[BTW, this last historical artifact makes me a bit angry, because at least one of the “cool” “search-based ‘SIEM’” vendors cannot do this today, in 2022, while the cheapest and simplest of the 1st generation SIM/SEMs could do it in 2003. This is literally 20 years of regress in front of your eyes!]

My writing at the time shows that I was quite obsessed with correlation, especially correlating normalized/categorized data that allowed creation of detection content (rules) without knowing the event IDs and other details from each log source [because single event matching was not considered cool in 2003, and so why is it acceptable in 2022?]

I have not kept copies of my oldest presentations (very few like This gem from 2003 survived), but mine old speaker site reminds me that I focused on Log Analysis for Security (2004), What Every Organization Should Monitor and Log (2005), and Log Mining for Security (2006).

The point I want to make with these fond memories is that many of the issues we faced in our security operations are the same issues we face today. The technological context has definitely changed, but the security challenges remain largely the same.

For example, Alert Congestion was the birth of SIM (typically Network IDS Alert Congestion at the time). Guess what problem we have argue now 20 years later? Alarm overload of course!

One of the things that I think the early SIEM did better than the later text-search based products was the focus on the quality of the data. In the early days, many naïve decisions were made (some for good reasons, some for bad, some due to technological limitations), e.g. For example, to focus on data that is considered security-sensitive and discard the rest. I remember agonizing over some Cisco event IDs when working with our log source integration lab.

Well, the more cynical among you might remind me that early SIEMs were plagued with parsing errors (yes, they were!) and that the data quality was – in that sense – inferior to that of a tool that collects raw logs. Unfortunately, that’s right (that made it hard Trust these SIEMs), so maybe this is a debate for another time. One approach I tried to use to solve it was to advocate the case for log standards.

XDR fans might find it awkward today, but the early SIEM pioneers really wanted to deliver effective and usable detection rules that are ready to use. We used to almost always call them correlation rules because we believe that to have a good product in space you need to have stateful correlation and not just simple matching.

On another topic, I was right early on in realizing that workflow is very important to a good SIEM product. This happened maybe 10 years before the first SOAR product was born. I remember designing a workflow system for our SIEM, assuming that people in the SOC will live in this UI most of the time…

In the early days, I also spent quite a bit of time looking at how threat actors behaved and then writing detection content (I even wrote ran a honeypot and then uses the observation to create correlation rules). Of course, that predated APT, so most of our recognition content was focused on commodity actors — script kiddies, as they were called back then. But hey – it wasn’t the auditors!

After a few years (circa 2006) I realized that a full collection of logs was going to be a thing and left my original SIEM employer. Note that I didn’t expect collecting raw logs to do this kill the original normalized, enriched and categorized SIEM… I wanted to have both full/raw and clean/enriched data (kinda what we have now…)

After that I left SIEM and had the opportunity to experience a very well built SaaS security tool – albeit not in the SIEM space. This got me excited about the ability to build a cloud-native or SaaS SIEM. sure, everyone knows that now. In 2009, SaaS SIEM didn’t really exist… but that’s a story for another day.

Well, I’m not sure I can be called an official SIEM historian (I may not be the oldest living SIEM developer, but it’s possible that I’m the one who has tolerated SIEM the longest in my life… ), but here is what I have.

Feel free to share your SIEM/SIM/SEM or Log Analysis Stories, Let us celebrate! 🙂

20 years of SIEM: I’m celebrating my dubious anniversary was originally published in Anton about security on Medium, where people continue the conversation by highlighting and responding to that story.

*** This is a syndicated blog from Security Bloggers Network Tales of Anton Chuvakin on Medium Written by Anton Chuvakin. Read the original post at: https://medium.com/anton-on-security/20-years-of-siem-celebrating-my-dubious-anniversary-f1cda2b453d3?source=rss-11065c9e943e——2


About Author

Comments are closed.