Organisations around the world are embracing cloud solutions to do business but managing data and assets remotely can quickly become a challenge.. Anomaly detection is a growing area of interest. In cloud systems, anomaly detection is the identification of rare events which could be indicative of critical failures or breach in the systems. This research, conducted by the SFI ADAPT Centre and Huawei, has the potential to reduce the substantial manpower needed to supervise machine learning methods for anomaly detection, and increase the accuracy of unsupervised methods. The novel approach aims to develop highly accurate supervised Artificial Intelligence (AI) driven approaches to anomaly detection that scale across multiple domains with minimal data labeling needed.
The results of the project were recently published in one of the top ranked AI journals worldwide, Knowledge Based Systems, and was led by ADAPT Professor John D. Kelleher and his team, along with research engineers Matthew Nicholsona and Rahul Agraharia, and Huawei Research Ireland’s Dr. Haythem Assem.
AI-driven anomaly detection can detect critical events in data sets such as intrusion detection, fraud, unwanted errors, along with opportunities. Speaking about the research, Professor Kelleher said: “Scaling modern AI solutions across an enterprise typically requires a huge investment in manual data labelling. This investment grows exponentially when the task for the AI is to identify rare events, such as anomalies. This project looked for solutions to this challenge by using multi-source transfer learning, a type of machine learning that reuses data from multiple related domains and tasks to develop bespoke solutions. We are really excited by our results which have the ability to advance capabilities and improve efficiencies in anomaly detection.”
The research explores the interaction between data normalization, the process of structuring data to reduce information redundancy and increase data integrity, and clustering, the process of grouping objects together based on similarity. These processes are examined in relation to a specific data area such as that of the Central Processing Unit (CPU). Within multi-source transfer learning, which is the utilization of information learned to carry out a task in one domain being used to aid learning in another domain.
The research findings show that supervised multi-source transfer learning outperform unsupervised methods on anomaly detection, and the results in this most recent paper highlight that to maximize system performance it is important to consider the interaction between normalisation and clustering, and also to consider whether sub-domains should be defined at the outset..
Cloud infrastructure and the reliability of those infrastructures are increasing in importance as we become more dependent on them. These infrastructures are now the standard platform for deploying software, which consequently affects the success of many technology companies and services including tech giants we rely on everyday such as Zoom, Huawei and Salesforce, and even companies working with cloud infrastructure in the Healthcare industry like Pfizer who utilised cloud systems to accelerate the development of a lung cancer drug.
This exciting collaborative project between ADAPT and Huawei Research Ireland, continues to advance the use of machine learning / deep learning to streamline efficiency in cloud systems, playing an important role in the future possibilities of cloud infrastructure and monitoring.
‘The interaction of normalisation and clustering in sub-domain definition for multi-source transfer learning based time series anomaly detection’ is available Online.
