Time series models predominantly, over the years, have focussed on individual time series via local models. This changed with the popularisation of deep learning techniques. This was also supported by the increase of temporal data availability, which led to many deep learning-based time series algorithms.

Due to their natural temporal ordering, time-series data are present in almost every task that is registered, taking into account some notion of ordering. From electronic health records and human activity recognition to acoustic scene classification and cyber-security, time series is encountered in many real-world applications.

Here are a few top works that improved the way we do time series modelling using deep learning:

Diverse Beam Search

Year: 2016

By Virginia Tech and Indiana University, USA

Beam search (BS) is widely used as an approximate inference algorithm to decode output sequences from neural sequence models. BS explores the search space in a greedy left-right fashion retaining only the top-B candidate, resulting in sequences that differ only slightly from each other.

To overcome this problem, the authors propose a Diverse Beam Search. DBS decodes a list of diverse outputs by optimising for a diversity-augmented objective. 

Moreover, these gains are achieved with minimal computational or memory overhead as compared to the beam search. The experiments were carried out on image captioning, machine translation and visual question generation using both standard quantitative metrics and qualitative human studies. The results show that this method consistently outperformed BS and previous techniques.

Distributed and Parallel Time Series Feature Extraction 

Year: 2016

By: Karlsruhe/ University of Auckland/ University of Freiburg

Feature selection is very challenging, especially for time series classification, for which each label or regression target is associated with several time-series and meta-information simultaneously.

This work presents an efficient, scalable feature extraction algorithm for time series, which filters the available features in an early stage of machine learning pipelines with respect to their significance for the classification or regression task while controlling the expected percentage of selected but irrelevant features. 

The proposed algorithm combines established feature extraction methods with a feature importance filter. It has low computational complexity and can work with only limited domain knowledge available.

ShallowRNN

Year: 2019

By: Microsoft

To induce long-term dependencies, and yet admit parallelisation, ShallowRNN was introduced. In this architecture, the first layer splits the input sequence and runs several independent RNNs. The second layer consumes the output of the first layer using a second RNN, thus capturing long dependencies.

Furthermore, the authors show that for time-series classification, this technique leads to substantially improved inference time over standard RNNs without compromising accuracy. For example, we can deploy audio-keyword classification on tiny Cortex M4 devices (TinyML), which was not possible using standard RNN models.

Multivariate LSTM-FCNs

Year: 2018

By: University Of Illinois, Chicago, USA

Over the past decade, multivariate time series classification has received great attention. We propose transforming the existing univariate time series classification models, the Long Short Term Memory Fully Convolutional Network (LSTM-FCN) and Attention LSTM-FCN (ALSTM-FCN), into a multivariate time series classification model by augmenting the fully convolutional block with a squeeze-and-excitation block to further improve accuracy. 

These models outperform most state-of-the-art models while requiring minimum preprocessing. The proposed models work efficiently on various complex multivariate time series classification tasks such as activity recognition or action recognition. Furthermore, the proposed models are highly efficient at test time and small enough to deploy on memory-constrained systems.

SOM-VAE

Year: 2019

By: ETH, Zurich, Switzerland

Representation learning in the context of time series data is usually difficult to interpret. Their non-intuitive nature comes from their high dimensional nature, which is not suitable for human understanding.

To address this problem, a new representation learning framework was proposed by the researchers at ETH, Zurich — self-organising maps and variational autoencoders. This framework allows one to learn discrete representations of time series.

We introduce a new way to overcome the non-differentiability in discrete representation learning and present a gradient-based version of the traditional self-organising map algorithm that is more performant than the original. 

GluonTS

Year: 2019

By: AWS

Introduced by cloud giant, Amazon web services, Gluon Time Series is a library for deep-learning-based time series modelling. It simplifies the experimentation with time series models for forecasting or anomaly detection. It has all necessary components for quickly building new models, for efficiently running and evaluating model accuracy.

Check more models and leaderboards here.