Beautiful Soup Primer: How To Scrape Data From A Website

Beautiful Soup is a HTML and XML parser available on Python 2.6+. Soup is named after the unstructured HTML documents which are hard to understand and noisy. It parses the data from the HTML and XML documents from where it can be extracted.

In this article, we will be going through functions which help us extract data from the HTML document. We will be using a toy HTML to explain how Beautiful Soup works and walk through the steps involved in Scraping — one of the techniques of data mining — data from a website’s HTML format.

With the help of headless browsers such as Selenium and PhanthomJS, one can easily practice how to scrape data out of a website. With these browsers, it will be easy to scrape through multiple pages or extract a large amount of data from the websites. Using a headless browser will also increase the computation speed which will result in the consumption of less memory. In fact, PhanthomJS assigns unique processes to each browser as well.

Subscribe to our Newsletter

Join our editors every weekday evening as they steer you through the most significant news of the day, introduce you to fresh perspectives, and provide unexpected moments of joy
Your newsletter subscriptions are subject to AIM Privacy Policy and Terms and Conditions.

Installing Beautiful Soup 4

Beautiful Soup library can be installed using PIP with a very simple command. It is available on almost all platforms. Here is a way to install it using Jupyter Notebook.

We can import this library with the following code and assign it to an object.

Installing An Alternative Parser

Beautiful Soup has a default parser available in the standard Python Library. We can use a different parser depending on the objective. The most common alternative parsers are “lxml” and “html5lib”. This can be installed with the help of the following code:

Below is a tabular representation of various parsers with their advantages and disadvantages:

Getting Started

We will be using this basic, and default, HTML doc to parse the data using Beautiful Soup.

The following code will expand HTML into its hierarchy:

Exploring The Parse Tree

To navigate through the tree, we can use the following commands:

Beautiful Soup has many attributes which can be accessed and edited. This extracted parsed data can be saved onto a text file.

To extract the text from the string, we can use the get_text() command.

Strings: How To Remove White spaces

The string can be accessed using the strings command. But it also includes white space which can be stripped easily.

Since the above output has a lot of white space, the striped.strings command will help us remove it.

Parent And Siblings

We can obtain the parent of a particular HTML with .parent attribute, like here:

To access the siblings — previous as well as the next — we can use the following commands:

Find And FindAll

This function is used to search for a very particular field throughout the HTML document. It is one of the key features required while data mining or scraping a data from a website with the help of Selenium and PhanthomJS.


Since finding the right tags from the HTML source is hard, scraping the data takes a lot of time. It can also depend on the amount of data extracted from a page. That is why, the wait time is necessary for the browser to load the data. Depending on their computation speed and availability of resources one can scrape data from almost any website using the right tools.

Kishan Maladkar
Kishan Maladkar holds a degree in Electronics and Communication Engineering, exploring the field of Machine Learning and Artificial Intelligence. A Data Science Enthusiast who loves to read about the computational engineering and contribute towards the technology shaping our world. He is a Data Scientist by day and Gamer by night.

Download our Mobile App

MachineHack | AI Hackathons, Coding & Learning

Host Hackathons & Recruit Great Data Talent!

AIMResearch Pioneering advanced AI market research

With a decade of experience under our belt, we are transforming how businesses use AI & data-driven insights to succeed.

The Gold Standard for Recognizing Excellence in Data Science and Tech Workplaces

With Best Firm Certification, you can effortlessly delve into the minds of your employees, unveil invaluable perspectives, and gain distinguished acclaim for fostering an exceptional company culture.

AIM Leaders Council

World’s Biggest Community Exclusively For Senior Executives In Data Science And Analytics.

3 Ways to Join our Community

Telegram group

Discover special offers, top stories, upcoming events, and more.

Discord Server

Stay Connected with a larger ecosystem of data science and ML Professionals

Subscribe to our Daily newsletter

Get our daily awesome stories & videos in your inbox