Time series models predominantly, over the years, have focussed on individual time series via local models. This changed with the popularisation of deep learning techniques. This was also supported by the increase of temporal data availability, which led to many deep learning-based time series algorithms.
Due to their natural temporal ordering, time-series data are present in almost every task that is registered, taking into account some notion of ordering. From electronic health records and human activity recognition to acoustic scene classification and cyber-security, time series is encountered in many real-world applications.
Here are a few top works that improved the way we do time series modelling using deep learning:
Subscribe to our Newsletter
Join our editors every weekday evening as they steer you through the most significant news of the day, introduce you to fresh perspectives, and provide unexpected moments of joy
Your newsletter subscriptions are subject to AIM Privacy Policy and Terms and Conditions.
Diverse Beam Search
Year: 2016
By Virginia Tech and Indiana University, USA
Beam search (BS) is widely used as an approximate inference algorithm to decode output sequences from neural sequence models. BS explores the search space in a greedy left-right fashion retaining only the top-B candidate, resulting in sequences that differ only slightly from each other.
To overcome this problem, the authors propose a Diverse Beam Search. DBS decodes a list of diverse outputs by optimising for a diversity-augmented objective.
Moreover, these gains are achieved with minimal computational or memory overhead as compared to the beam search. The experiments were carried out on image captioning, machine translation and visual question generation using both standard quantitative metrics and qualitative human studies. The results show that this method consistently outperformed BS and previous techniques.
Distributed and Parallel Time Series Feature Extraction
Year: 2016
By: Karlsruhe/ University of Auckland/ University of Freiburg
Feature selection is very challenging, especially for time series classification, for which each label or regression target is associated with several time-series and meta-information simultaneously.
This work presents an efficient, scalable feature extraction algorithm for time series, which filters the available features in an early stage of machine learning pipelines with respect to their significance for the classification or regression task while controlling the expected percentage of selected but irrelevant features.
The proposed algorithm combines established feature extraction methods with a feature importance filter. It has low computational complexity and can work with only limited domain knowledge available.
ShallowRNN
Year: 2019
By: Microsoft
To induce long-term dependencies, and yet admit parallelisation, ShallowRNN was introduced. In this architecture, the first layer splits the input sequence and runs several independent RNNs. The second layer consumes the output of the first layer using a second RNN, thus capturing long dependencies.
Furthermore, the authors show that for time-series classification, this technique leads to substantially improved inference time over standard RNNs without compromising accuracy. For example, we can deploy audio-keyword classification on tiny Cortex M4 devices (TinyML), which was not possible using standard RNN models.
Multivariate LSTM-FCNs
Year: 2018
By: University Of Illinois, Chicago, USA
Over the past decade, multivariate time series classification has received great attention. We propose transforming the existing univariate time series classification models, the Long Short Term Memory Fully Convolutional Network (LSTM-FCN) and Attention LSTM-FCN (ALSTM-FCN), into a multivariate time series classification model by augmenting the fully convolutional block with a squeeze-and-excitation block to further improve accuracy.
These models outperform most state-of-the-art models while requiring minimum preprocessing. The proposed models work efficiently on various complex multivariate time series classification tasks such as activity recognition or action recognition. Furthermore, the proposed models are highly efficient at test time and small enough to deploy on memory-constrained systems.
SOM-VAE
Year: 2019
By: ETH, Zurich, Switzerland
Representation learning in the context of time series data is usually difficult to interpret. Their non-intuitive nature comes from their high dimensional nature, which is not suitable for human understanding.
To address this problem, a new representation learning framework was proposed by the researchers at ETH, Zurich — self-organising maps and variational autoencoders. This framework allows one to learn discrete representations of time series.
We introduce a new way to overcome the non-differentiability in discrete representation learning and present a gradient-based version of the traditional self-organising map algorithm that is more performant than the original.
GluonTS
Year: 2019
By: AWS
Introduced by cloud giant, Amazon web services, Gluon Time Series is a library for deep-learning-based time series modelling. It simplifies the experimentation with time series models for forecasting or anomaly detection. It has all necessary components for quickly building new models, for efficiently running and evaluating model accuracy.
Check more models and leaderboards here.
