Why Machine Learning Is The Right Fit For Anomaly Detection: Interview With Siddharth Bhatia

“Anomaly detection has great significance in detecting fake profiles in Social Networks like Twitter, Facebook, Amazon reviews, and even financial frauds.”

For this week’s ML practitioner’s series, Analytics India Magazine got in touch with Siddharth Bhatia, who is into machine learning research at National University of Singapore (NUS). His work focuses majorly on Streaming Anomaly Detection. At NUS, he is supported by a President’s Graduate Fellowship. He has also been previously recognised as a young researcher in the ACM Heidelberg Laureate Forum. In this interview, Bhatia will talk about his research and shares a few tips for beginners.

AIM:How important is a PhD in ML?

Siddharth: After completing the masters in Computer Science from BITS Pilani, I started my Ph.D. in Streaming Anomaly Detection to detect anomalies in real-time so as to minimise the harm caused by them. For someone wanting to go into academia or even a research lab in the industry, PhD is kind of a necessity, though it is possible that someone does good quality research in ML/AI without a PhD. ML research is not for a select few but the masses. Also, there are a lot of misconceptions caused because of the hype.

AIM: What made you curious about anomaly detection?

Siddharth: Anomaly detection is a critical problem for finding suspicious behaviour in many systems. Intrusion detection, fake ratings, and financial transaction fraud are few of the many examples. That being said, anomaly detection has been well researched over the years — the majority of the approaches proposed in anomaly detection focus on static graphs. However, real-world graphs are usually dynamic in nature, and current approaches can risk missing temporal characteristics of the graphs and anomalies due to their dynamic nature.

AIM Daily XO

Join our editors every weekday evening as they steer you through the most significant news of the day, introduce you to fresh perspectives, and provide unexpected moments of joy
Your newsletter subscriptions are subject to AIM Privacy Policy and Terms and Conditions.

Coming to dynamic graphs, most of them aggregate edges into graph snapshots. To recover from malicious activities as soon as possible, we need them in real-time or near real-time. 

Fraudulent or anomalous events typically occur in microclusters. In other words, they occur in groups of suspiciously similar edges. Considering, the number of vertices can increase as we process the stream of edges, we require an algorithm which leverages constant memory in the graph size. Existing methods that process edge streams in an online manner aim to detect individually surprising edges, not microclusters, and can thus miss large amounts of suspicious activity. 

Download our Mobile App

To address this, we proposed MIDAS (Microcluster-Based Detector of Anomalies in Edge Streams). The goal here is to detect microcluster anomalies. Additionally, by using a principled hypothesis testing framework, MIDAS provides theoretical bounds on the false positive probability, unlike previous methods.

AIM: Can you tell us about your research methodology?

Siddharth: Our approach, MIDAS, finds anomalous edges from a dynamic graph in a streaming manner. The idea here is to combine a chi-squared goodness-of-fit test with the Count-Min-Sketch (CMS) streaming data structures. This, in turn, would get an anomaly score for each edge. Our method also incorporates temporal and spatial relations to achieve better performance.

For instance, recent intrusion detection datasets typically report tens of features such as its source and destination IP, protocol, average packet size, etc. It is important to design approaches that can handle multi-aspect data. In the intrusion detection setting, MIDAS treats all variables of the dataset as categorical variables, whereas we also want to handle arbitrary mixtures of categorical variables (e.g. source IP address) and numerical variables (e.g. average packet size). We, therefore, propose MSTREAM, a method for processing a stream of multi-aspect data like event-logs and multi-attributed graphs, that detects group anomalies and incorporates correlations between the features. 

For our research, we have mainly used Python for coding; sci-kit learn library frequently. And, in the case of frameworks, I prefer PyTorch. Whereas AWS happens to be my go-to cloud option. 

AIM: What are the key findings from your research?

Siddharth: MIDAS uses unsupervised machine learning to detect anomalies in a streaming manner in real-time. This approach was designed to address the recent sophisticated attacks. Our approach can be used to detect intrusions, Denial of Service (DoS), Distributed Denial of Service (DDoS) attacks, financial fraud, and fake ratings. MIDAS provides theoretical guarantees on the false positives and is three orders of magnitude faster than the existing state of the art solutions. 

The findings of our work on MIDAS can be summarised as follows:

  • We could detect microcluster anomalies or suspiciously similar edges.
  • We give theoretical guarantees on the false positive probability.
  • MIDAS is independent of the graph size.
  • MIDAS allows for real-time anomaly detection.
  • It is scalable and can process up to 4 million edges in less than 1 second on a normal laptop.
  • It is up to 48% more accurate and 644 times faster than the state of the art approaches.

AIM: How do you plan on taking this research forward?

Siddharth: We have open-sourced MIDAS source code and different implementations and are available on the Github project page. So far, many developers have implemented MIDAS in Python, Ruby, Rust, R, and Golang in addition to the C++ version we originally released. 

MIDAS is currently being deployed in real-world systems to improve their performance. We have been approached by cybersecurity firms too.

We have extended MIDAS to MStream where we detect anomalies on high dimensional multi-aspect data having both categorical and numeric attributes. In terms of both accuracy and running time, MStream outperformed several baselines including popular scikit-learn algorithms like Isolation Forest and Local Outlier Factor. 

MIDAS can also be used to detect fake profiles in Social Networks like Twitter, Facebook, Amazon reviews, and Financial Frauds. 

AIM: Can you recommend some resources to know more about this field?

Siddharth: There are some nice surveys which give an extensive overview of the field:

  • “Graph-based anomaly detection and description: a survey” by Akoglu, Leman, Hanghang Tong, and Danai Koutra.
  • “Tensor-based anomaly detection: An interdisciplinary survey” by Fanaee-T, Hadi, and Joao Gama. 
  • “Deep learning for anomaly detection: A survey” by Chalapathy, Raghavendra, and Sanjay Chawla.

Sign up for The Deep Learning Podcast

by Vijayalakshmi Anandan

The Deep Learning Curve is a technology-based podcast hosted by Vijayalakshmi Anandan - Video Presenter and Podcaster at Analytics India Magazine. This podcast is the narrator's journey of curiosity and discovery in the world of technology.

Ram Sagar
I have a master's degree in Robotics and I write about machine learning advancements.

Our Upcoming Events

24th Mar, 2023 | Webinar
Women-in-Tech: Are you ready for the Techade

27-28th Apr, 2023 I Bangalore
Data Engineering Summit (DES) 2023

23 Jun, 2023 | Bangalore
MachineCon India 2023 [AI100 Awards]

21 Jul, 2023 | New York
MachineCon USA 2023 [AI100 Awards]

3 Ways to Join our Community

Telegram group

Discover special offers, top stories, upcoming events, and more.

Discord Server

Stay Connected with a larger ecosystem of data science and ML Professionals

Subscribe to our Daily newsletter

Get our daily awesome stories & videos in your inbox

Council Post: Evolution of Data Science: Skillset, Toolset, and Mindset

In my opinion, there will be considerable disorder and disarray in the near future concerning the emerging fields of data and analytics. The proliferation of platforms such as ChatGPT or Bard has generated a lot of buzz. While some users are enthusiastic about the potential benefits of generative AI and its extensive use in business and daily life, others have raised concerns regarding the accuracy, ethics, and related issues.