Active Hackathon

Beautiful Soup Primer: How To Scrape Data From A Website

Beautiful Soup is a HTML and XML parser available on Python 2.6+. Soup is named after the unstructured HTML documents which are hard to understand and noisy. It parses the data from the HTML and XML documents from where it can be extracted.

In this article, we will be going through functions which help us extract data from the HTML document. We will be using a toy HTML to explain how Beautiful Soup works and walk through the steps involved in Scraping — one of the techniques of data mining — data from a website’s HTML format.


Sign up for your weekly dose of what's up in emerging technology.

With the help of headless browsers such as Selenium and PhanthomJS, one can easily practice how to scrape data out of a website. With these browsers, it will be easy to scrape through multiple pages or extract a large amount of data from the websites. Using a headless browser will also increase the computation speed which will result in the consumption of less memory. In fact, PhanthomJS assigns unique processes to each browser as well.

Installing Beautiful Soup 4

Beautiful Soup library can be installed using PIP with a very simple command. It is available on almost all platforms. Here is a way to install it using Jupyter Notebook.

We can import this library with the following code and assign it to an object.

Installing An Alternative Parser

Beautiful Soup has a default parser available in the standard Python Library. We can use a different parser depending on the objective. The most common alternative parsers are “lxml” and “html5lib”. This can be installed with the help of the following code:

Below is a tabular representation of various parsers with their advantages and disadvantages:

Getting Started

We will be using this basic, and default, HTML doc to parse the data using Beautiful Soup.

The following code will expand HTML into its hierarchy:

Exploring The Parse Tree

To navigate through the tree, we can use the following commands:

Beautiful Soup has many attributes which can be accessed and edited. This extracted parsed data can be saved onto a text file.

To extract the text from the string, we can use the get_text() command.

Strings: How To Remove White spaces

The string can be accessed using the strings command. But it also includes white space which can be stripped easily.

Since the above output has a lot of white space, the striped.strings command will help us remove it.

Parent And Siblings

We can obtain the parent of a particular HTML with .parent attribute, like here:

To access the siblings — previous as well as the next — we can use the following commands:

Find And FindAll

This function is used to search for a very particular field throughout the HTML document. It is one of the key features required while data mining or scraping a data from a website with the help of Selenium and PhanthomJS.


Since finding the right tags from the HTML source is hard, scraping the data takes a lot of time. It can also depend on the amount of data extracted from a page. That is why, the wait time is necessary for the browser to load the data. Depending on their computation speed and availability of resources one can scrape data from almost any website using the right tools.

More Great AIM Stories

Kishan Maladkar
Kishan Maladkar holds a degree in Electronics and Communication Engineering, exploring the field of Machine Learning and Artificial Intelligence. A Data Science Enthusiast who loves to read about the computational engineering and contribute towards the technology shaping our world. He is a Data Scientist by day and Gamer by night.

Our Upcoming Events

Conference, Virtual
Genpact Analytics Career Day
3rd Sep

Conference, in-person (Bangalore)
Cypher 2022
21-23rd Sep

Conference, in-person (Bangalore)
Machine Learning Developers Summit (MLDS) 2023
19-20th Jan

Conference, in-person (Bangalore)
Data Engineering Summit (DES) 2023
21st Apr, 2023

3 Ways to Join our Community

Discord Server

Stay Connected with a larger ecosystem of data science and ML Professionals

Telegram Channel

Discover special offers, top stories, upcoming events, and more.

Subscribe to our newsletter

Get the latest updates from AIM
How Data Science Can Help Overcome The Global Chip Shortage

China-Taiwan standoff might increase Global chip shortage

After Nancy Pelosi’s visit to Taiwan, Chinese aircraft are violating Taiwan’s airspace. The escalation made TSMC’s chairman go public and threaten the world with consequences. Can this move by China fuel a global chip shortage?

Another bill bites the dust

The Bill had faced heavy criticism from different stakeholders -citizens, tech firms, political parties since its inception

So long, Spotify

‘TikTok Music’ is set to take over the online streaming space, but there exists an app that has silently established itself in the Indian market.