Guide To Detrending Using Scipy Signal

In time-series data, cyclic or non-cyclic, an increment or decrement in the magnitude of any variable over time is called the trend of the variable. As mentioned, trends can be cyclic or non-cyclic. If the trend is non-cyclic, it can cause accuracy loss in prediction.

There are many kinds of trends we can identify, but the simplest types can be classified are:

• Trend behaviour. 
• Dataset behaviour. 

Trend behaviour:

• Deterministic trend:  In this type of trend, there is consistency, either decremental or incremental.
• Stochastic trend: In this type of trend, there is inconsistency, either decremental or incremental.

Dataset behaviour:

• Global trend: Trend which is present in the whole dataset.
• Local trend: Trend which is present in only a part or many parts of the dataset.

What is Detrending?

Removal of trends from a time series data set is called detrend. It allows us to know about the normal cyclic trend and the trend beyond the cycle. Detrending can be done using many statistical techniques such as quadratic detrending, linear trending, Baxter-King filter etc.

Detrending time series provides us with normal cyclic time series data as it removes non-cyclic trends. This process tells about the factors affecting the time series data and finds important factors about it. There are many examples of detrending; one can be the removal of unwanted or unpredicted changes(natural disasters) in weather forecasting datasets to forecast basic weather in a particular season. 

As we have discussed, there are many techniques and ways to detrend a time series data. In machine learning, trends of data can represent two requirements:

• Removal of information because it can vary the relationship between input and output variables. 
• Addition of information to improve the relationship between input and output variables. 

This is a very common practice to remove the trends when using statistical methods for forecasting, but it does not always help in improving prediction when using machine learning models.

Removal or addition of information or trends can be useful for improving the results.

Next, in this article, we will learn about the removal of trends using scipy signal, and we will see how it can be helpful to make a trend normalised using python.

Code Implementation for Detrending 

Parameter of scipy.signal.detrend:

 scipy.signal.detrend(data, axis=- 1, type='linear', bp=0, overwrite_data=False)

Data: array, input data.

Axis: int, optional

By default, this is set as the last axis(-1)

Type : {‘linear’ , ‘constant’}

If type == ‘linear’(default), linear least square of the data is subtracted from the data.

If type == ‘constant’, the mean of the data is subtracted from the data.

bp: array_like of ints, optional 

A sequence of breakpoints. If given, an individual linear fit is performed for each part of data between two breakpoints. 

This parameter only effects when type == ‘linear’.

Overwrite data: bool, optional

Default is false; if true, perform detrending and avoiding a copy.

Setup environment in colab:

Requirements: Python 3.6 or above, Pandas 1.2.5, NumPy 1.21.0, scipy 1.7.0 or above 

Importing the required libraries :

 from scipy import signal
 import numpy as np
 from numpy.random import default_rng
 import pandas as pd
 from pandas import datetime
 from matplotlib import pyplot 

Generating data using Numpy Linspace :

Input :

 rng = default_rng()
 npoints = 1000
 noise = rng.standard_normal(npoints)
 x = 3 + 2*np.linspace(0, 1, npoints) + noise
 x 

Output:

In these codes, we have called Random Generator to normalise noise in the data set for deviation in trend and added it to x to make it a rough trend; we will see it in visualisation.

Checking the maximum difference between x and Noise. 

Input:

(x-noise).max()

Output:

Normalising it using Scipy Signal.

Input:

(signal.detrend(x) - noise).max()

Output :

Here we can see after detrending x, the maximum difference between the x and noise is significantly lower than before.

Let’s visualise the situation using trend graphs.

Making graphs for noise:

Input :

 pyplot.plot(noise)
 pyplot.show() 

Output:

Making graphs for x series :

Input :

 pyplot.plot(x)
 pyplot.show() 

Output:

Visualising the trend of x after detrending it. 

Input :

 y=(signal.detrend(x) - noise)
 pyplot.plot(y)
 pyplot.show() 

Output: 

From the graph, we can see the effect of detrending in the x series. Before detrending, we have seen that there was an abnormal trend in the data. Therefore, we have removed that abnormal trend from the series.

Till now, we have seen how detrending with Scipy Signal. Next, we will see detrending on shampoo-sales data. So let’s take a look at the data and its values.

Importing the shampoo-sales data set.

Input :

series = pd.read_csv('shampoo-sales .csv', header=0, parse_dates=[0], index_col=0, squeeze=True)
 series.head(10) 

Output:

In the above output, we can see a monthly column with their sales value in the sales column. First, let’s see the trend in the data set.

Input :

 series.plot()
 pyplot.show() 

Output: 

Detrend by scipy signal.

Remove linear trend axis from data(type = ‘linear’)

Input :

 x = series.values
 y=(signal.detrend(x, axis=-1, type='linear', bp=0))
 y 

Output:

Making trend graph of y array.

Input :

 pyplot.plot(y)
 pyplot.show() 

Output:

Detrending by subtracting the mean of data from the data(type = ‘constant’).

Input:

 x = series.values
 y=(signal.detrend(x, axis=-1, type='constant', bp=0))
 y 

Output:

Making a trend graph of y array.

Input:

 pyplot.plot(y)
 pyplot.show() 

Output:

In the above output, we can see how the scipy signal detrend works, the data set shows an increasing trend. We detrend the data set with two methods of scipy signal detrend. One or both approaches may be the solution to our problem.

Let’s compare all the graphs we generated after using the scipy signal detrend method.

 In the first detrend method, we can see that it has effectively detrended the data set. But there is a parabola structure in the residual. In that case, a polynomial method can be fruitful.

This article shows how detrending is useful for making the right predictions in data and discussed scipy signal to detrend a time series data set and some basic classification of trends.

References

All the information written in this article is gathered from:

• Scipy Signal detrend.
• Colab notebook for codes. 
• Shampoo-sales dataset.