# Guide To Detrending Using Scipy Signal

In time-series data, cyclic or non-cyclic, an increment or decrement in the magnitude of any variable over time is called the trend of the variable. As mentioned, trends can be cyclic or non-cyclic. If the trend is non-cyclic, it can cause accuracy loss in prediction.

There are many kinds of trends we can identify, but the simplest types can be classified are:

• Trend behaviour.
• Dataset behaviour.

Trend behaviour:

• Deterministic trend:  In this type of trend, there is consistency, either decremental or incremental.
• Stochastic trend: In this type of trend, there is inconsistency, either decremental or incremental.

Dataset behaviour:

• Global trend: Trend which is present in the whole dataset.
• Local trend: Trend which is present in only a part or many parts of the dataset.

## What is Detrending?

Removal of trends from a time series data set is called detrend. It allows us to know about the normal cyclic trend and the trend beyond the cycle. Detrending can be done using many statistical techniques such as quadratic detrending, linear trending, Baxter-King filter etc.

Detrending time series provides us with normal cyclic time series data as it removes non-cyclic trends. This process tells about the factors affecting the time series data and finds important factors about it. There are many examples of detrending; one can be the removal of unwanted or unpredicted changes(natural disasters) in weather forecasting datasets to forecast basic weather in a particular season.

As we have discussed, there are many techniques and ways to detrend a time series data. In machine learning, trends of data can represent two requirements:

• Removal of information because it can vary the relationship between input and output variables.
• Addition of information to improve the relationship between input and output variables.

This is a very common practice to remove the trends when using statistical methods for forecasting, but it does not always help in improving prediction when using machine learning models.

Removal or addition of information or trends can be useful for improving the results.

Next, in this article, we will learn about the removal of trends using scipy signal, and we will see how it can be helpful to make a trend normalised using python.

## Code Implementation for Detrending

Parameter of scipy.signal.detrend:

` scipy.signal.detrend(data, axis=- 1, type='linear', bp=0, overwrite_data=False)`

Data: array, input data.

Axis: int, optional

By default, this is set as the last axis(-1)

Type : {‘linear’ , ‘constant’}

If type == ‘linear’(default), linear least square of the data is subtracted from the data.

If type == ‘constant’, the mean of the data is subtracted from the data.

bp: array_like of ints, optional

A sequence of breakpoints. If given, an individual linear fit is performed for each part of data between two breakpoints.

This parameter only effects when type == ‘linear’.

Overwrite data: bool, optional

Default is false; if true, perform detrending and avoiding a copy.

Setup environment in colab:

Requirements: Python 3.6 or above, Pandas 1.2.5, NumPy 1.21.0, scipy 1.7.0 or above

Importing the required libraries :

``` from scipy import signal
import numpy as np
from numpy.random import default_rng
import pandas as pd
from pandas import datetime
from matplotlib import pyplot ```

Generating data using Numpy Linspace :

Input :

``` rng = default_rng()
npoints = 1000
noise = rng.standard_normal(npoints)
x = 3 + 2*np.linspace(0, 1, npoints) + noise
x ```

Output:

In these codes, we have called Random Generator to normalise noise in the data set for deviation in trend and added it to x to make it a rough trend; we will see it in visualisation.

Checking the maximum difference between x and Noise.

Input:

`(x-noise).max()`

Output:

Normalising it using Scipy Signal.

Input:

`(signal.detrend(x) - noise).max()`

Output :

Here we can see after detrending x, the maximum difference between the x and noise is significantly lower than before.

Let’s visualise the situation using trend graphs.

Making graphs for noise:

Input :

``` pyplot.plot(noise)
pyplot.show() ```

Output:

Making graphs for x series :

Input :

``` pyplot.plot(x)
pyplot.show() ```

Output:

Visualising the trend of x after detrending it.

Input :

``` y=(signal.detrend(x) - noise)
pyplot.plot(y)
pyplot.show() ```

Output:

From the graph, we can see the effect of detrending in the x series. Before detrending, we have seen that there was an abnormal trend in the data. Therefore, we have removed that abnormal trend from the series.

Till now, we have seen how detrending with Scipy Signal. Next, we will see detrending on shampoo-sales data. So let’s take a look at the data and its values.

Importing the shampoo-sales data set.

Input :

```series = pd.read_csv('shampoo-sales .csv', header=0, parse_dates=[0], index_col=0, squeeze=True)

Output:

In the above output, we can see a monthly column with their sales value in the sales column. First, let’s see the trend in the data set.

Input :

``` series.plot()
pyplot.show() ```

Output:

Detrend by scipy signal.

Remove linear trend axis from data(type = ‘linear’)

Input :

``` x = series.values
y=(signal.detrend(x, axis=-1, type='linear', bp=0))
y ```

Output:

Making trend graph of y array.

Input :

``` pyplot.plot(y)
pyplot.show() ```

Output:

Detrending by subtracting the mean of data from the data(type = ‘constant’).

Input:

``` x = series.values
y=(signal.detrend(x, axis=-1, type='constant', bp=0))
y ```

Output:

Making a trend graph of y array.

Input:

``` pyplot.plot(y)
pyplot.show() ```

Output:

In the above output, we can see how the scipy signal detrend works, the data set shows an increasing trend. We detrend the data set with two methods of scipy signal detrend. One or both approaches may be the solution to our problem.

Let’s compare all the graphs we generated after using the scipy signal detrend method.

In the first detrend method, we can see that it has effectively detrended the data set. But there is a parabola structure in the residual. In that case, a polynomial method can be fruitful.

This article shows how detrending is useful for making the right predictions in data and discussed scipy signal to detrend a time series data set and some basic classification of trends.

## References

Yugesh is a graduate in automobile engineering and worked as a data analyst intern. He completed several Data Science projects. He has a strong interest in Deep Learning and writing blogs on data science and machine learning.

## Our Upcoming Events

### Telegram group

Discover special offers, top stories, upcoming events, and more.

### Discord Server

Stay Connected with a larger ecosystem of data science and ML Professionals

### Is MongoDB Vector Search the Panacea for all LLM Problems?

By introducing proprietary data, developers can narrow down the pool of possible responses, significantly reducing the likelihood of hallucinations

### Why Intel Closing the Gap with NVIDIA is Good News

Gaudi2’s performance surpassed NVIDIA H100’s on a state-of-the-art vision language model on Hugging Face’s performance benchmarks

### Can OpenAI Save SoftBank?

After a tumultuous investment spree with significant losses, will SoftBank’s plans to invest in OpenAI and other AI companies provide the boost it needs?

### Oracle’s Grand Multicloud Gamble

“Cloud Should be Open,” says Larry at Oracle CloudWorld 2023, Las Vegas, recollecting his discussions with Microsoft chief Satya Nadella last week.

### NVIDIA Catches Up to AMD, Intel with MCM Design

GH100 was also expected to have an MCM design, but it came with a monolithic architecture again.

### 13 Startups that Google Funded Under its First AI Startup Cohort

Google’s inaugural AI Startup Cohort features thirteen diverse startups that address important issues with AI. The 10-week program offers tailored support, mentoring, and technical expertise

### Unity is Not as United as It Sounds

Unity’s new pricing model has caused an exodus to Unreal and Godot. Developers give up on Unity, accusing it of being anti-consumer

### KPI Partners: Accelerating Business Growth with Modernization and Monetization of Data Assets

KPI Partners, a leader in digital transformation, is dedicated to accelerate business profitability by providing customized data analytics and digital transformation services and solutions to address the unique needs of each client.

### How Generative AI is Revolutionising Data Science Tools

Einblick Prompt enables users to create complete data workflows using natural language, accelerating various stages of data science and analytics. Einblick has effectively combined the capabilities of a Jupyter notebook with the user-friendliness of ChatGPT.

### Google is Officially Killing the Internet with AI

While Google has long preached ‘helpful content written by people, for people, in search results’, its recent actions suggest otherwise