ZetCode

Python BeautifulSoup

last modified October 30, 2026 Python 3.15

In this article we show how to use the BeautifulSoup Python library. The examples find tags, traverse the document tree, modify the document, and scrape web pages.

BeautifulSoup

BeautifulSoup is a Python library for parsing HTML and XML documents. It is often used for web scraping. BeautifulSoup transforms a complex HTML document into a tree of Python objects, such as tags, navigable strings, and comments.

Installing BeautifulSoup

BeautifulSoup is installed from the beautifulsoup4 package; it is imported as bs4. We also install the lxml parser, which BeautifulSoup uses in our examples, and the requests library for downloading web pages.

$ pip install beautifulsoup4 lxml requests

Parsers

BeautifulSoup does not parse the document itself; it uses a parser. The parser is chosen with the second argument of the BeautifulSoup constructor.

For valid documents, the parsers produce the same tree. They differ in how they fix broken HTML.

parsers.py
from bs4 import BeautifulSoup

broken = '<p>An <b>unclosed tag<li>item</p>'

for parser in ['html.parser', 'lxml', 'html5lib']:
    try:
        soup = BeautifulSoup(broken, parser)
        print(f'{parser:12} {soup}')
    except Exception as e:
        print(f'{parser:12} {type(e).__name__}: {e}')

We parse a fragment with an unclosed b tag and an li element outside a list. The lxml parser adds the html and body elements; html5lib builds the same tree as a browser would. It is a good idea to always name the parser explicitly, so that the program behaves the same on every computer.

$ python parsers.py
html.parser  <p>An <b>unclosed tag<li>item</li></b></p>
lxml         <html><body><p>An <b>unclosed tag<li>item</li></b></p></body></html>
html5lib     <html><head></head><body><p>An <b>unclosed tag</b></p><li><b>item<p></p></b></li></body></html>

The HTML file

In most examples, we will use the following HTML file:

index.html
<!DOCTYPE html>
<html>
    <head>
        <title>Header</title>
        <meta charset="utf-8">
    </head>

    <body>
        <h2>Operating systems</h2>

        <ul id="mylist" style="width:150px">
            <li>Solaris</li>
            <li>FreeBSD</li>
            <li>Debian</li>
            <li>NetBSD</li>
            <li>Windows</li>
        </ul>

        <p>
          FreeBSD is an advanced computer operating system used to
          power modern servers, desktops, and embedded platforms.
        </p>

        <p>
          Debian is a Unix-like computer operating system that is
          composed entirely of free software.
        </p>

    </body>
</html>

The document has a heading, a list with an id, and two paragraphs.

Python BeautifulSoup simple example

In the first example, we use the BeautifulSoup module to get three tags.

simple.py
from bs4 import BeautifulSoup

with open('index.html', encoding='utf-8') as f:
    contents = f.read()

soup = BeautifulSoup(contents, 'lxml')

print(soup.h2)
print(soup.head)
print(soup.li)

The code example prints the HTML code of three tags.

from bs4 import BeautifulSoup

We import the BeautifulSoup class from the bs4 module. The BeautifulSoup is the main class for doing work.

with open('index.html', encoding='utf-8') as f:
    contents = f.read()

We open the index.html file and read its contents with the read method.

soup = BeautifulSoup(contents, 'lxml')

A BeautifulSoup object is created; the HTML data is passed to the constructor. The second argument specifies the parser.

print(soup.h2)
print(soup.head)

Here we print the HTML code of two tags: h2 and head. Accessing a tag name as an attribute returns the first tag of that name.

print(soup.li)

There are multiple li elements; the line prints the first one.

$ python simple.py
<h2>Operating systems</h2>
<head>
<title>Header</title>
<meta charset="utf-8"/>
</head>
<li>Solaris</li>

BeautifulSoup tags, name, text

The name attribute of a tag gives its name and the text attribute its text content.

tags_names.py
from bs4 import BeautifulSoup

with open('index.html', encoding='utf-8') as f:
    soup = BeautifulSoup(f, 'lxml')

print(f'HTML: {soup.h2}, name: {soup.h2.name}, text: {soup.h2.text}')
print(type(soup.h2))
print(type(soup.h2.string))

The code example prints the HTML code, name, and text of the h2 tag. The constructor also accepts an open file, so we do not have to read it first.

print(type(soup.h2))
print(type(soup.h2.string))

Tags are Tag objects. The text inside a tag is a NavigableString, a subclass of str.

$ python tags_names.py
HTML: <h2>Operating systems</h2>, name: h2, text: Operating systems
<class 'bs4.element.Tag'>
<class 'bs4.element.NavigableString'>

BeautifulSoup traverse tags

With the descendants attribute of the soup object, we traverse the whole HTML document.

traverse_tree.py
from bs4 import BeautifulSoup

with open('index.html', encoding='utf-8') as f:
    soup = BeautifulSoup(f, 'lxml')

for el in soup.descendants:
    if el.name:
        print(el.name)

The example goes through the document tree and prints the names of all HTML tags. Strings have no name, so the if condition skips them. The older recursiveChildGenerator method does the same, but it is deprecated.

$ python traverse_tree.py
html
head
title
meta
body
h2
ul
li
li
li
li
li
p
p

BeautifulSoup element children

With the children attribute, we can get the direct children of a tag.

get_children.py
from bs4 import BeautifulSoup

with open('index.html', encoding='utf-8') as f:
    soup = BeautifulSoup(f, 'lxml')

root = soup.html

print([e.name for e in root.children])
print([e.name for e in root.children if e.name is not None])
print([e.name for e in soup.ul.find_all(recursive=False)])

The example retrieves the children of the html tag.

print([e.name for e in root.children])

The children attribute also returns the whitespace strings between the tags; their name is None.

print([e.name for e in root.children if e.name is not None])

We add a condition to include only the tags.

print([e.name for e in soup.ul.find_all(recursive=False)])

Alternatively, find_all with recursive=False returns only the child tags.

$ python get_children.py
[None, 'head', None, 'body', None]
['head', 'body']
['li', 'li', 'li', 'li', 'li']

BeautifulSoup element descendants

With the descendants attribute we get all descendants (children of all levels) of a tag.

get_descendants.py
from bs4 import BeautifulSoup

with open('index.html', encoding='utf-8') as f:
    soup = BeautifulSoup(f, 'lxml')

root = soup.body

print([e.name for e in root.descendants if e.name is not None])

The example retrieves all descendants of the body tag.

$ python get_descendants.py
['h2', 'ul', 'li', 'li', 'li', 'li', 'li', 'p', 'p']

BeautifulSoup parents and siblings

We can also move up the tree and sideways. The parent and parents attributes return the enclosing tags; the find_next_sibling and find_previous_sibling methods find tags at the same level.

navigate.py
from bs4 import BeautifulSoup

with open('index.html', encoding='utf-8') as f:
    soup = BeautifulSoup(f, 'lxml')

debian = soup.find('li', string='Debian')

print(debian.parent.name, debian.parent['id'])
print([p.name for p in debian.parents])

print(debian.find_previous_sibling('li'))
print(debian.find_next_sibling('li'))
print([li.text for li in debian.find_next_siblings('li')])

print(soup.ul.find_next_sibling('p').text.split()[0])

We start from the li tag that contains Debian.

print([p.name for p in debian.parents])

The parents generator goes up to the soup object itself, whose name is [document].

print(soup.ul.find_next_sibling('p').text.split()[0])

The first paragraph after the list is its next sibling of the p type. We print its first word.

$ python navigate.py
ul mylist
['ul', 'body', 'html', '[document]']
<li>FreeBSD</li>
<li>NetBSD</li>
['NetBSD', 'Windows']
FreeBSD

BeautifulSoup attributes

The attributes of a tag are accessed like dictionary items. For the next examples, we use a second HTML file with a table and links.

products.html
<!DOCTYPE html>
<html lang="en">
<head>
    <meta charset="utf-8">
    <title>Products</title>
</head>
<body>
    <h1>Products</h1>

    <table id="products">
        <thead>
            <tr><th>Name</th><th>Price</th><th>Stock</th></tr>
        </thead>
        <tbody>
            <tr class="product"><td><a href="/products/1">Keyboard</a></td><td>49.90</td><td>12</td></tr>
            <tr class="product sale"><td><a href="/products/2">Mouse</a></td><td>19.50</td><td>0</td></tr>
            <tr class="product"><td><a href="/products/3">Monitor</a></td><td>189.00</td><td>4</td></tr>
        </tbody>
    </table>

    <p>
        See also <a href="https://www.example.com/catalog" title="Full catalog">the full catalog</a>
        and <a href="docs/shipping.html">shipping information</a>.
    </p>
</body>
</html>

The table has an id, the rows have classes, and one link has a title.

attributes.py
from bs4 import BeautifulSoup

with open('products.html', encoding='utf-8') as f:
    soup = BeautifulSoup(f, 'lxml')

table = soup.table
print(table['id'])
print(table.attrs)

row = soup.find('tr', class_='sale')
print(row['class'])

link = soup.find('a', title=True)
print(link.attrs)
print(link.get('title'))
print(link.get('target'))
print(link.get('target', '_self'))

try:
    print(link['target'])
except KeyError as e:
    print(f'KeyError: {e}')

We read the attributes of several tags.

row = soup.find('tr', class_='sale')
print(row['class'])

Since class is a Python keyword, the argument is called class_. The class attribute can have multiple values, so BeautifulSoup returns it as a list.

link = soup.find('a', title=True)
print(link.attrs)

The title=True argument finds a tag that has the attribute with any value. The attrs dictionary contains all attributes.

print(link.get('target'))
print(link.get('target', '_self'))

The get method returns None or a default value for a missing attribute, while the square brackets raise a KeyError.

$ python attributes.py
products
{'id': 'products'}
['product', 'sale']
{'href': 'https://www.example.com/catalog', 'title': 'Full catalog'}
Full catalog
None
_self
KeyError: 'target'

BeautifulSoup get text

The text attribute returns all the text inside a tag, including the whitespace from the HTML source. The get_text method and the stripped_strings generator give us more control.

get_text.py
from bs4 import BeautifulSoup

with open('index.html', encoding='utf-8') as f:
    soup = BeautifulSoup(f, 'lxml')

ul = soup.ul

print(repr(ul.text))
print(ul.get_text(', ', strip=True))
print(list(ul.stripped_strings))

p = soup.p
print(' '.join(p.text.split()))

We get the text of the list in several ways.

print(ul.get_text(', ', strip=True))

The get_text method joins the strings with the given separator; strip=True removes the surrounding whitespace from each string and drops the empty ones.

print(' '.join(p.text.split()))

To normalize the whitespace inside a paragraph, we split the text into words and join them with single spaces.

$ python get_text.py
'\nSolaris\nFreeBSD\nDebian\nNetBSD\nWindows\n'
Solaris, FreeBSD, Debian, NetBSD, Windows
['Solaris', 'FreeBSD', 'Debian', 'NetBSD', 'Windows']
FreeBSD is an advanced computer operating system used to power modern servers, desktops, and embedded platforms.

BeautifulSoup find elements by id

With the find method we can find elements by various means including element id. It returns the first match or None.

find_by_id.py
from bs4 import BeautifulSoup

with open('index.html', encoding='utf-8') as f:
    soup = BeautifulSoup(f, 'lxml')

print(soup.find('ul', id='mylist'))
print(soup.find('ul', attrs={'id': 'mylist'}) is soup.find(id='mylist'))
print(soup.find('ol'))

The code example finds the ul tag that has the mylist id.

print(soup.find('ul', attrs={'id': 'mylist'}) is soup.find(id='mylist'))

Attributes can be given as keyword arguments or in the attrs dictionary; the latter works for attributes whose names are not valid Python identifiers, such as data-id. The tag name is optional.

print(soup.find('ol'))

There is no ol tag in the document, so find returns None.

$ python find_by_id.py
<ul id="mylist" style="width:150px">
<li>Solaris</li>
<li>FreeBSD</li>
<li>Debian</li>
<li>NetBSD</li>
<li>Windows</li>
</ul>
True
None

BeautifulSoup find all tags

With the find_all method we can find all elements that meet some criteria.

find_all.py
from bs4 import BeautifulSoup

with open('index.html', encoding='utf-8') as f:
    soup = BeautifulSoup(f, 'lxml')

for tag in soup.find_all('li'):
    print(f'{tag.name}: {tag.text}')

print(soup.find_all('li', limit=2))

The code example finds and prints all li tags. The limit argument stops the search after the given number of matches.

$ python find_all.py
li: Solaris
li: FreeBSD
li: Debian
li: NetBSD
li: Windows
[<li>Solaris</li>, <li>FreeBSD</li>]

Finding multiple tag types

The find_all method can take a list of tag names to search for.

find_all2.py
from bs4 import BeautifulSoup

with open('index.html', encoding='utf-8') as f:
    soup = BeautifulSoup(f, 'lxml')

tags = soup.find_all(['h2', 'p'])

for tag in tags:
    print(f"{tag.name}: {' '.join(tag.text.split())}")

The example finds all h2 and p elements and prints their text. The elements are returned in document order.

$ python find_all2.py
h2: Operating systems
p: FreeBSD is an advanced computer operating system used to power modern servers, desktops, and embedded platforms.
p: Debian is a Unix-like computer operating system that is composed entirely of free software.

Finding tags with a function

The find_all method can also take a function which determines which elements should be returned. The function receives a tag and returns a boolean value.

find_by_fun.py
from bs4 import BeautifulSoup


def is_bsd_item(tag):
    return tag.name == 'li' and tag.text.endswith('BSD')


with open('index.html', encoding='utf-8') as f:
    soup = BeautifulSoup(f, 'lxml')

print(soup.find_all(is_bsd_item))
print(soup.find_all(lambda tag: tag.is_empty_element))

The is_bsd_item function selects list items that end with BSD.

print(soup.find_all(lambda tag: tag.is_empty_element))

A lambda selects empty elements. The only empty element in the document is meta.

$ python find_by_fun.py
[<li>FreeBSD</li>, <li>NetBSD</li>]
[<meta charset="utf-8"/>]

Finding with regular expressions

It is also possible to find elements and strings by using regular expressions.

regex.py
import re

from bs4 import BeautifulSoup

with open('index.html', encoding='utf-8') as f:
    soup = BeautifulSoup(f, 'lxml')

strings = soup.find_all(string=re.compile('BSD'))

for txt in strings:
    print(' '.join(txt.split()))

print([tag.name for tag in soup.find_all(re.compile('^h[1-6]$|^ul$'))])

With the string argument, find_all searches the text of the document and returns the matching strings. We print the strings that contain BSD.

print([tag.name for tag in soup.find_all(re.compile('^h[1-6]$|^ul$'))])

A regular expression in place of the tag name matches tag names: here, headings and unordered lists.

$ python regex.py
FreeBSD
NetBSD
FreeBSD is an advanced computer operating system used to power modern servers, desktops, and embedded platforms.
['h2', 'ul']

BeautifulSoup CSS selectors

With the select and select_one methods, we can use CSS selectors to find elements. The select method returns a list of all matches, select_one the first match or None.

css_selectors.py
from bs4 import BeautifulSoup

with open('index.html', encoding='utf-8') as f:
    soup = BeautifulSoup(f, 'lxml')

print(soup.select('li:nth-of-type(3)'))
print(soup.select_one('li:nth-of-type(3)'))
print(soup.select_one('#mylist').name)
print([li.text for li in soup.select('ul#mylist > li:nth-child(odd)')])
print([li.text for li in soup.select('li:-soup-contains("BSD")')])
print(len(soup.select('body p')))

This example uses several CSS selectors.

print(soup.select('li:nth-of-type(3)'))
print(soup.select_one('li:nth-of-type(3)'))

Both lines find the third li element; the first one returns it in a list.

print(soup.select_one('#mylist').name)

The # character is used in CSS to select tags by their id attribute.

print([li.text for li in soup.select('ul#mylist > li:nth-child(odd)')])
print([li.text for li in soup.select('li:-soup-contains("BSD")')])

The > combinator selects direct children, and :nth-child(odd) every other item. The :-soup-contains pseudo-class is a non-standard extension that matches elements containing the given text.

$ python css_selectors.py
[<li>Debian</li>]
<li>Debian</li>
ul
['Solaris', 'Debian', 'Windows']
['FreeBSD', 'NetBSD']
2

Selecting by class and attribute

CSS selectors work well with classes and attributes. We use the products.html file.

select_products.py
from bs4 import BeautifulSoup

with open('products.html', encoding='utf-8') as f:
    soup = BeautifulSoup(f, 'lxml')

print([a.text for a in soup.select('tr.product a')])
print([a.text for a in soup.select('tr.sale a')])
print([a['href'] for a in soup.select('a[href^="http"]')])
print(soup.select_one('tbody tr:last-child td:nth-of-type(2)').text)

The tr.product a selector finds links in rows with the product class, and tr.sale a those in rows with the sale class.

print([a['href'] for a in soup.select('a[href^="http"]')])

The ^= attribute selector matches values with the given prefix, so we get the absolute links.

print(soup.select_one('tbody tr:last-child td:nth-of-type(2)').text)

We get the price, the second cell of the last row.

$ python select_products.py
['Keyboard', 'Mouse', 'Monitor']
['Mouse']
['https://www.example.com/catalog']
189.00

Extracting table data

A common scraping task is to turn an HTML table into Python data.

table_data.py
from bs4 import BeautifulSoup

with open('products.html', encoding='utf-8') as f:
    soup = BeautifulSoup(f, 'lxml')

table = soup.find('table', id='products')

headers = [th.text.lower() for th in table.thead.find_all('th')]

products = []

for row in table.tbody.find_all('tr'):
    values = [td.get_text(strip=True) for td in row.find_all('td')]
    product = dict(zip(headers, values))

    product['price'] = float(product['price'])
    product['stock'] = int(product['stock'])
    products.append(product)

for product in products:
    print(product)

in_stock = sum(p['price'] * p['stock'] for p in products)
print(f'stock value: {in_stock:.2f}')

We read the table into a list of dictionaries.

headers = [th.text.lower() for th in table.thead.find_all('th')]

The column names come from the th cells of the table head.

values = [td.get_text(strip=True) for td in row.find_all('td')]
product = dict(zip(headers, values))

For each row of the table body, we pair the cell texts with the column names. The text of a cell includes the text of its link.

product['price'] = float(product['price'])
product['stock'] = int(product['stock'])

All values from HTML are strings; we convert the numbers so that we can compute the stock value.

$ python table_data.py
{'name': 'Keyboard', 'price': 49.9, 'stock': 12}
{'name': 'Mouse', 'price': 19.5, 'stock': 0}
{'name': 'Monitor', 'price': 189.0, 'stock': 4}
stock value: 1354.80

Extracting links

Links in a page are often relative. The urljoin function from urllib.parse resolves them against the URL of the page.

links.py
from urllib.parse import urljoin

from bs4 import BeautifulSoup

base = 'https://shop.example.com/store/'

with open('products.html', encoding='utf-8') as f:
    soup = BeautifulSoup(f, 'lxml')

for a in soup.find_all('a', href=True):
    print(f'{a.text:24} {urljoin(base, a["href"])}')

The href=True argument finds only a tags that have an href attribute.

print(f'{a.text:24} {urljoin(base, a["href"])}')

Links starting with / are resolved against the host, other relative links against the directory of the page; absolute links are kept.

$ python links.py
Keyboard                 https://shop.example.com/products/1
Mouse                    https://shop.example.com/products/2
Monitor                  https://shop.example.com/products/3
the full catalog         https://www.example.com/catalog
shipping information     https://shop.example.com/store/docs/shipping.html

BeautifulSoup append element

The append method appends a new tag to the HTML document.

append_tag.py
from bs4 import BeautifulSoup

with open('index.html', encoding='utf-8') as f:
    soup = BeautifulSoup(f, 'lxml')

newtag = soup.new_tag('li')
newtag.string = 'OpenBSD'

ultag = soup.ul
ultag.append(newtag)

print(ultag.prettify())

The example appends a new li tag.

newtag = soup.new_tag('li')
newtag.string = 'OpenBSD'

First, we create a new tag with the new_tag method and set its text.

ultag = soup.ul
ultag.append(newtag)

We get the reference to the ul tag and append the newly created tag to it.

print(ultag.prettify())

We print the ul tag in a neat format.

$ python append_tag.py
<ul id="mylist" style="width:150px">
 <li>
  Solaris
 </li>
 <li>
  FreeBSD
 </li>
 <li>
  Debian
 </li>
 <li>
  NetBSD
 </li>
 <li>
  Windows
 </li>
 <li>
  OpenBSD
 </li>
</ul>

BeautifulSoup insert element

The insert method inserts a tag at the specified position.

insert_tag.py
from bs4 import BeautifulSoup

with open('index.html', encoding='utf-8') as f:
    soup = BeautifulSoup(f, 'lxml')

newtag = soup.new_tag('li', attrs={'class': 'new'})
newtag.string = 'OpenBSD'

ultag = soup.ul
ultag.insert(2, newtag)

print([li.text for li in ultag.find_all('li')])
print(newtag.find_previous_sibling('li'))

The example creates a li tag with a class attribute and inserts it into the ul tag.

ultag.insert(2, newtag)

The position counts all children, including the whitespace strings between the tags. Position 2 is after the first li element and the newline that follows it, so the new item becomes the second one.

$ python insert_tag.py
['Solaris', 'OpenBSD', 'FreeBSD', 'Debian', 'NetBSD', 'Windows']
<li>Solaris</li>

BeautifulSoup replace text

The replace_with method replaces an element or a string with another one.

replace_text.py
from bs4 import BeautifulSoup

with open('index.html', encoding='utf-8') as f:
    soup = BeautifulSoup(f, 'lxml')

tag = soup.find(string='Windows')
tag.replace_with('OpenBSD')

h2 = soup.h2
h2.string = 'Free operating systems'

print(h2)
print(soup.ul.prettify())

The example finds a specific string with the find method and replaces it with the replace_with method.

h2.string = 'Free operating systems'

Assigning to the string attribute replaces the whole content of a tag.

$ python replace_text.py
<h2>Free operating systems</h2>
<ul id="mylist" style="width:150px">
 <li>
  Solaris
 </li>
 <li>
  FreeBSD
 </li>
 <li>
  Debian
 </li>
 <li>
  NetBSD
 </li>
 <li>
  OpenBSD
 </li>
</ul>

BeautifulSoup remove element

The decompose method removes a tag from the tree and destroys it.

decompose_tag.py
from bs4 import BeautifulSoup

with open('index.html', encoding='utf-8') as f:
    soup = BeautifulSoup(f, 'lxml')

ptag2 = soup.select_one('p:nth-of-type(2)')
ptag2.decompose()

for li in soup.find_all('li', string=['Windows', 'Solaris']):
    li.decompose()

del soup.ul['style']

print(soup.body.prettify())

The example removes the second p element.

for li in soup.find_all('li', string=['Windows', 'Solaris']):
    li.decompose()

We also remove two list items. The string argument accepts a list of strings.

del soup.ul['style']

Attributes are removed with the del statement.

$ python decompose_tag.py
<body>
 <h2>
  Operating systems
 </h2>
 <ul id="mylist">
  <li>
   FreeBSD
  </li>
  <li>
   Debian
  </li>
  <li>
   NetBSD
  </li>
 </ul>
 <p>
  FreeBSD is an advanced computer operating system used to
          power modern servers, desktops, and embedded platforms.
 </p>
</body>

BeautifulSoup prettify code

With the prettify method, we can make the HTML code look better.

prettify.py
from bs4 import BeautifulSoup

html = '<html><head><title>Hello</title></head><body><p>Today is a <b>beautiful</b> day.<p>Hello there.</body></html>'

soup = BeautifulSoup(html, 'lxml')

print(soup.prettify())

We prettify the HTML code of a small document written on a single line. Each tag and string is placed on its own line, and the second paragraph is closed by the parser.

$ python prettify.py
<html>
 <head>
  <title>
   Hello
  </title>
 </head>
 <body>
  <p>
   Today is a
   <b>
    beautiful
   </b>
   day.
  </p>
  <p>
   Hello there.
  </p>
 </body>
</html>

BeautifulSoup web scraping

To scrape a web page, we first download it. Requests is a simple Python HTTP library. It provides methods for accessing web resources via HTTP.

scraping.py
import requests
from bs4 import BeautifulSoup

resp = requests.get('https://example.com', timeout=10)
resp.raise_for_status()

soup = BeautifulSoup(resp.text, 'lxml')

print(soup.title)
print(soup.title.text)
print(soup.title.parent.name)
print(soup.p.text)

The example retrieves the title of a simple web page and the name of the title's parent tag. It also prints the text of the first paragraph.

resp = requests.get('https://example.com', timeout=10)
resp.raise_for_status()

soup = BeautifulSoup(resp.text, 'lxml')

We get the HTML data of the page. The raise_for_status method raises an exception if the server returns an error status, so we do not try to parse an error page.

$ python scraping.py
<title>Example Domain</title>
Example Domain
head
This domain is for use in documentation examples without needing permission. This is not a service; avoid relying on it for testing and monitoring purposes.

BeautifulSoup scraping with built-in web server

We can also serve HTML pages with the built-in HTTP server of Python, which is useful for testing scrapers. See Python http.server.

scraping_local.py
import requests
from bs4 import BeautifulSoup

resp = requests.get('http://localhost:8000/', timeout=10)

soup = BeautifulSoup(resp.text, 'lxml')

print(soup.title.text)
print([li.text for li in soup.select('#mylist li')])

The program gets the document from the locally running server and prints its title and list items.

We create a public directory, copy the index.html file there, and start the server in the background. The server logs the request of the scraper. Finally, we stop the server with Ctrl+C.

$ mkdir public
$ cp index.html public/
$ python -m http.server --bind 127.0.0.1 --directory public 8000 &
Serving HTTP on 127.0.0.1 port 8000 (http://127.0.0.1:8000/) ...
$ python scraping_local.py
Header
['Solaris', 'FreeBSD', 'Debian', 'NetBSD', 'Windows']
127.0.0.1 - - [30/Oct/2026 18:48:31] "GET / HTTP/1.1" 200 -

Keyboard interrupt received, exiting.

Source

Beautiful Soup documentation

In this article we have worked with the Python BeautifulSoup library.

Author

I am Jan Bodnar, a programmer and technical writer. I have been writing programming tutorials at ZetCode since 2007 and have published eight e-books. See about the author.

List all Python tutorials.