How to Write a Python Broken Link Checker: Find Dead Links With requests and BeautifulSoup

CodexFuse Team · 2026-11-06T03:20:36.737Z · 21 min read min read

How to Write a Python Broken Link Checker: Find Dead Links With requests and BeautifulSoup

A python broken link checker is a small script that crawls a web page, collects every link on it, and tests each one to find the dead ends. This guide builds one with requests and BeautifulSoup, the two libraries nearly every Python tutorial reaches for, and the finished script is under 80 lines long. You will fetch a page, extract its links, check each status code, and write a CSV report you can act on.

Before you write any code, it helps to know what a healthy link looks like from the other side. A SHA-256 hash generator runs entirely in your browser and never touches a server, which is the opposite of the checker you are about to build: yours will make a real request to every link it finds. That is also why the checker needs manners, which the later sections cover.

This python script to find broken links is aimed at developers and site owners who run their own site, so it assumes you can run Python on your machine or on a server. VPS hosting gives you a full Linux server to run scripts like this on a schedule, and if you would rather skip the code entirely, the broken link checker on CodexFuse does the same job in your browser with no setup at all.

What Is a Python Broken Link Checker?

A python broken link checker is a program that finds links on a web page that no longer lead anywhere. It fetches a page, pulls out every link, asks each destination for its status code, and reports the ones that answer 404, 410, or another error, or do not answer at all. A link is broken when the server says the page is gone, the server itself errors, or the request fails before any answer arrives.

The checker in this guide does that job for one page at a time, and the sitemap section later extends it to a whole site. It uses requests to fetch pages and make the status checks, and BeautifulSoup to read the HTML and pull out the links. The combination people search for, check broken links python requests beautifulsoup, is exactly this pair: requests does the talking, and BeautifulSoup does the reading.

Broken links matter because they waste the reader's time and the search engine's crawl budget, and they quietly drain the value of the page that links out. Seotal's guide, published June 25, 2026, puts it plainly: a short Python script with requests and BeautifulSoup can find broken links automatically, saving hours of manual checking. The script below is that idea, built out and tested on a real page.

What Do You Need Before You Start?

You need Python 3.10 or newer, the requests library, and the BeautifulSoup library, plus a text editor and a terminal. Install both libraries with two pip commands, and the commands below were tested on Python 3.12.3 with requests 2.34.2 and beautifulsoup4 4.15.0, the current versions as of November 6, 2026.

Run pip install requests and then pip install beautifulsoup4, each on its own line. The package is called beautifulsoup4 on PyPI, and the import in your code is from bs4 import BeautifulSoup. The old BeautifulSoup package is the long-dead version 3, and the docs warn that it is not what you want.

The two libraries are close in age as well as in purpose: requests 2.34.2 shipped on May 14, 2026, and beautifulsoup4 4.15.0 followed 24 days later, on June 7. Requests is the most downloaded HTTP library in the Python world, with PyPI reporting around 300 million downloads a week, which works out to roughly 43 million a day. That popularity is why nearly every broken-link tutorial reaches for the same two libraries.

How Does the Checker Work in Five Steps?

The find broken links python workflow is the same in every tutorial worth reading: fetch, extract, resolve, check, report. Any broken link checker python guide covers the same five steps, and Seotal's six-step workflow and Chandan Kumar's three methods on DEV are the same ground seen from different angles. The five steps below are the whole script, and the sections after this one explain each step in turn.

1. Fetch the target page with requests.get and a 10-second timeout.
2. Parse the HTML with BeautifulSoup and collect every link with find_all("a", href=True).
3. Turn each link into a full URL with urljoin, strip fragments, and keep only http and https links.
4. Check each URL with a HEAD request, falling back to GET when the server refuses HEAD, and record the status code.
5. Print the broken links and write the full results to a CSV file.

That is the entire design. The rest of this guide is the code for each step, the edge cases that trip beginners up, and a live run that shows the script working on a real page.

Here is the complete script, exactly as it was run for the live test in the final section. Save it as broken_link_checker.py and run it as python3 broken_link_checker.py https://example.com report.csv, and it prints the broken links and writes the full report to the CSV file.

"""Find broken links on one web page.

Usage: python3 broken_link_checker.py https://example.com [output.csv]
Requires: pip install requests beautifulsoup4
"""
import csv
import sys
from concurrent.futures import ThreadPoolExecutor, as_completed
from urllib.parse import urldefrag, urljoin, urlparse

import requests
from bs4 import BeautifulSoup

TIMEOUT = 10
MAX_WORKERS = 10
HEADERS = {"User-Agent": "Mozilla/5.0 (compatible; PythonBrokenLinkChecker/1.0)"}


def extract_links(page_url):
    """Fetch one page and return its de-duplicated http(s) links."""
    resp = requests.get(page_url, headers=HEADERS, timeout=TIMEOUT)
    resp.raise_for_status()
    soup = BeautifulSoup(resp.text, "html.parser")
    links = set()
    for tag in soup.find_all("a", href=True):
        href = tag["href"].strip()
        if href.startswith(("mailto:", "tel:", "javascript:", "#")):
            continue
        full, _ = urldefrag(urljoin(page_url, href))
        if urlparse(full).scheme in ("http", "https"):
            links.add(full)
    return sorted(links)


def check_link(url):
    """Return (url, status, error). HEAD first; GET fallback on 405/403."""
    try:
        resp = requests.head(url, headers=HEADERS, allow_redirects=True, timeout=TIMEOUT)
        if resp.status_code in (405, 403):
            resp = requests.get(url, headers=HEADERS, allow_redirects=True,
                                timeout=TIMEOUT, stream=True)
            resp.close()
        return url, resp.status_code, None
    except requests.RequestException as exc:
        return url, None, str(exc)


def main():
    if len(sys.argv) < 2:
        print("Usage: python3 broken_link_checker.py <URL> [output.csv]")
        sys.exit(2)
    page_url = sys.argv[1]
    out_csv = sys.argv[2] if len(sys.argv) > 2 else "broken_links.csv"

    links = extract_links(page_url)
    print(f"Found {len(links)} links on {page_url}")
    results = []
    with ThreadPoolExecutor(max_workers=MAX_WORKERS) as pool:
        futures = {pool.submit(check_link, url): url for url in links}
        for future in as_completed(futures):
            results.append(future.result())

    broken = [(u, s, e) for u, s, e in results if s is None or s >= 400]
    for url, status, error in sorted(broken):
        print(f"[BROKEN] {status or error}  {url}")
    print(f"\nChecked {len(links)} links: {len(broken)} broken.")

    with open(out_csv, "w", newline="", encoding="utf-8") as fh:
        writer = csv.writer(fh)
        writer.writerow(["url", "status_code", "error"])
        for url, status, error in sorted(results):
            writer.writerow([url, status if status is not None else "", error or ""])
    print(f"Full report written to {out_csv}")


if __name__ == "__main__":
    main()

How Do You Fetch a Page and Extract Its Links?

You fetch the page with requests.get, pass the page text to BeautifulSoup with the html.parser parser, and ask for every anchor tag that has an href attribute. The docs' own quick-start example does exactly this: soup.find_all('a') returns every anchor, and link.get('href') reads the address out of each one. A 10-second timeout keeps a slow page from hanging your script forever, because requests does not time out unless you tell it to.

The parser matters more than it looks. BeautifulSoup works with three parsers, and the docs recommend lxml for speed when you can install it, while html.parser ships with Python and needs nothing extra. The table below compares the three, using the docs' own descriptions.

This guide uses html.parser because it needs no extra install and is fast enough for a link check. If you later point the script at thousands of pages, switching to lxml is a one-word change, and the rest of the code stays the same.

How Do You Turn Relative Links Into Full URLs?

You resolve each link against the page's own address with urllib.parse.urljoin, which turns a relative href such as FAQ.html into a full URL. The Python docs' own example shows urljoin('http://www.cwi.nl/%7Eguido/Python.html', 'FAQ.html') producing 'http://www.cwi.nl/%7Eguido/FAQ.html'. Absolute links pass through unchanged, so the same call handles both kinds.

Two cleanups keep the results honest. First, strip the fragment with urldefrag, so page.html#section and page.html count as one link instead of two.

Second, keep only links whose scheme is http or https, which drops mailto:, tel:, and javascript: links that can never be checked with a request. JMooreWV's crawler and Chandan Kumar's method 3 both apply the same filter, and the reference script follows them.

De-duplication is the third cleanup, and it matters more than the first two. A navigation menu repeats the same links on every page, so a set keeps each target once and the checker asks for it once. JMooreWV's script caches checked links the same way, and the reference script collects links into a set before it checks anything.

How to Check Broken Links Using Python: HEAD First, Then GET

You check a link by asking its server for a status code, and the cheapest way to ask is a HEAD request, which fetches headers only and never downloads the body. If the server answers 405 or 403, the script falls back to a GET request, because some servers reject HEAD even though the page itself is fine. The fallback is in the reference script, and Python Friday's guide recommends adding it for exactly this reason.

Requests does not follow redirects for HEAD requests by default, so the check passes allow_redirects=True and follows the chain to the final address. The docs state that by default requests performs location redirection for all verbs except HEAD, and that Response.history lists every hop along the way. A redirect is not a broken link, but a chain of three or more hops is worth fixing on your own site, because each hop costs time and can break later.

The verdict rule is simple, and every source fetched for this guide agrees on it: a status code of 400 or above means broken, a 2xx code means alive, and a 3xx code means the link moved and the redirect should be followed. Chandan Kumar states the rule verbatim, and the same rule runs every script in the research for this guide.

How Do You Check Hundreds of Links Without Waiting All Day?

You check them at the same time with concurrent.futures.ThreadPoolExecutor, because link checking spends most of its time waiting on the network. The script submits every link to a pool of ten workers and collects the results as they finish, using as_completed. Seotal and Kramer Labs both use ten workers, and Chandan Kumar's version uses twenty.

Ten workers is a starting point, not a law, and the right number depends on whose server you are hitting. Chandan Kumar warns that twenty concurrent requests to one domain can get you blocked, and Emir Mrkaljevic's sitemap audit keeps it to two or three requests at a time per site. Respect the site's robots.txt file and keep the rate polite, because a checker that floods one host looks exactly like a bot.

The arithmetic shows why it matters. At one second per check, 119 links take about two minutes one at a time, and ten workers cut that to roughly twelve seconds, because nine of the ten requests are waiting while one runs. The default pool size in Python is min(32, CPU count + 4), so an 8-core machine gets twelve workers unless you set the number yourself.

How Do You Save the Results as a CSV Report?

You write the results with Python's csv module, one row per link, with the URL, the status code, and any error message. Open the file with newline='' exactly as the docs require, or the writer will put a blank line after every row on Windows. The reference script writes every checked link, not just the broken ones, so the report doubles as proof of what was tested.

Printing the broken links to the terminal is the other half of the report, and the script prints each one as it finds it, then a one-line summary at the end. Seotal's guide adds CSV export and optional email alerts on top of the same printout, and a weekly cron run at 2 AM on Mondays is its suggested schedule. The CSV is the part that lasts: paste it into a spreadsheet, sort by status code, and the 404s and 410s sit at the top of the fix list.

How Do You Check a Whole Site With Its Sitemap?

You read the site's sitemap.xml, pull out every page URL it lists, and run the same check on each page in turn. Python Friday's guide treats the sitemap as the entry point, and JMooreWV's crawler adds a sitemap mode that expands sitemap indexes too. The sitemap is a list the site owner maintains, so it is the most complete list of pages you can get without crawling the site yourself.

If your site has no sitemap yet, a sitemap generator builds one from a list of URLs, and the checker can then read the file it produces. A sitemap check only sees what the sitemap lists, which is both its strength and its limit.

Orphaned pages that no sitemap entry mentions will never be tested, and links injected by JavaScript are invisible to it, because requests only sees the raw HTML. Robin Beraud's sitemap checker states both limits plainly, and Emir Mrkaljevic's version discovers the sitemap through robots.txt, which is the standard place to advertise it.

The upgrade path is straightforward: loop over the sitemap's URLs, run the single-page checker on each one, and write one combined CSV. That is the point where the script stops being a snippet and starts being a small project, because a whole-site run also needs a visited set, a page limit, and politeness delays. Chandan Kumar says the same thing about whole-site crawling: it needs a queue, a visited set, depth limits, and robots.txt handling, and that is a real project, not a snippet.

How Do You Read the Results Without Chasing False Alarms?

You read the status code first and the error second, because the code tells you whether the link is dead and the error tells you why the check itself failed. A 404 or a 410 is a real broken link: fix the href, redirect the target, or remove the link. A 500-series code means the server is broken or down, so re-check it later before you delete anything, because the link may come back on its own.

The false alarms are the 403s, 429s, and 999s. Kramer Labs found that LinkedIn, X, and Instagram often answer automated requests with 999, 403, or 520 even when the page works fine in a browser, and the right verdict for those is check by hand, not broken. A 429 means you asked too fast, so slow down and retry later instead of reporting the link as dead.

Redirects deserve their own line in the report. A 301 that lands on the right page is not broken, but the old address should still be updated, because every redirect is a hop that can break later and a small delay for the reader. W3C's link checker warns about redirects for the same reason, and a report that lists them separately from the errors keeps the fix list honest.

What Are the Limits of a Home-Grown Script?

A home-grown script is not a replacement for a hosted checker on a large site, and the sources for this guide agree on why. JavaScript-rendered pages hide their links from requests, because requests only sees the raw HTML, and the script has no way past a CAPTCHA or a bot-detection wall. Chandan Kumar lists all three limits, and Geekflare adds that the script has no code for proxies, so a scan at scale gets throttled or banned.

The W3C Link Checker shows the other end of the spectrum: a free service that has checked links since August 1998, which makes it about 28 years old as of November 2026. It sleeps at least one second between requests to each server and caps the number of documents it checks recursively, so it is slow by design and honest about it. The exact recursive cap is not stated in its docs, so do not quote a number for it.

That is where the CodexFuse tool fits. The free broken link checker tool post explains the no-code route, and the checker itself runs in your browser with no install, no scheduling, and no code to maintain.

Use the script when you want the logic in your own hands and a report you can shape, and use the tool when you want the answer without the setup. CodexFuse does not offer a Python broken-link-checker tool, and this guide does not promise one.

What Did the Script Find on a Real Page?

The reference script was run against a real page before this guide was published: the Python.org applications page, https://www.python.org/about/apps/, on November 6, 2026. It found 119 links on the page and checked every one of them with a 10-second timeout and ten workers. The run is the proof that the code in this guide works as printed, and the numbers below are the actual results, not an illustration.

Of the 119 links, 109 answered 200, one answered 404, one answered 403, and eight timed out. The 404 was http://wiki.python.org/moin/TkInter, a dead wiki page, and it is the same dead link Kramer Labs found independently on the same page in October 2026, when their run also counted 119 links. The 403 was a bot-block false alarm of the kind the previous section describes, and the eight timeouts went through this author's sandbox network proxy, so they are not confirmed broken links.

The same page gave the same count twice: Kramer Labs counted 119 links on it in October 2026, and this run counted 119 again in November. Their run found three broken links, the same 404 plus two connection-refused hosts, while this run found one confirmed dead link and eight timeouts that belong to the test network, not the page.

Geekflare's guide ran a similar script on the Stripe docs and found 47 links with one broken 404, about 2.1 percent, which is the same shape of result: most links alive, one or two dead. The shares in the table are rounded to one decimal place, which is why they add to 99.9 rather than 100.

Sources

Every fact in this guide comes from a source fetched on November 6, 2026. Library versions are the versions current on that date, and the live-run numbers are the results of a run made for this guide.

• Seotal, "How to use Python to Analyze Website Broken Links for SEO", published June 25, 2026: https://www.seotal.com/blog/python-to-analyze-website-broken-links
• Chandan Kumar, "How to Check Broken Links Using Python (3 Ways)", DEV, published June 9, 2026: https://dev.to/chandan-kumar/how-to-check-broken-links-using-python-3-ways-1d27
• Geekflare, "How to Find Broken Links: 4 Methods (SEO Tools, APIs, and Python)", July 21, 2026: https://geekflare.com/guides/how-to-find-broken-links/
• Kramer Labs, "How to Find Broken Links on Any Website, Free Python Script", DEV, October 31, 2026: https://dev.to/kramer_labs/how-to-find-broken-links-on-any-website-free-python-script-no-code-option-ji5
• Kasia Chil, "Build Your Own SEO Bot: How to Automate Broken Link Audits with Python", DEV, September 1, 2026: https://dev.to/kasia_chil_fa339fd2c4f786/build-your-own-seo-bot-how-to-automate-broken-link-audits-with-python-35o0
• Python Friday #140, "Create a Basic Link Checker", September 2022: https://pythonfriday.dev/2022/09/140-create-a-basic-link-checker/
• JMooreWV, "Building a Python Broken Link Checker": https://jmoorewv.com/guides/programming/python/building-a-python-broken-link-checker/
• Robin Beraud, "Find Broken Pages on Your Site From Its Sitemap With One Python File", DEV: https://dev.to/robinberaud/find-broken-pages-on-your-site-from-its-sitemap-with-one-python-file-557f
• Emir Mrkaljevic, "How to Find Broken Links and Redirects Hiding in Your XML Sitemap With Python", DEV, October 9, 2026: https://dev.to/emirmrkaljevic/how-to-find-broken-links-and-redirects-hiding-in-your-xml-sitemap-with-python-55ih
• requests on PyPI, version 2.34.2: https://pypi.org/project/requests/
• requests documentation, Quickstart and Advanced: https://requests.readthedocs.io/en/latest/user/quickstart/ and https://requests.readthedocs.io/en/latest/user/advanced/
• beautifulsoup4 on PyPI, version 4.15.0: https://pypi.org/project/beautifulsoup4/
• Beautiful Soup documentation (Crummy): https://www.crummy.com/software/BeautifulSoup/bs4/doc/
• Python documentation, concurrent.futures: https://docs.python.org/3/library/concurrent.futures.html
• Python documentation, csv: https://docs.python.org/3/library/csv.html
• Python documentation, urllib.parse: https://docs.python.org/3/library/urllib.parse.html
• MDN, HTTP response status codes: https://developer.mozilla.org/en-US/docs/Web/HTTP/Reference/Status
• W3C Link Checker service: https://validator.w3.org/checklink
• W3C Link Checker documentation: https://validator.w3.org/checklink/docs/checklink.html

Frequently Asked Questions

What libraries do I need for a Python broken link checker?

Two: requests and BeautifulSoup. Install them with pip install requests and pip install beautifulsoup4. The setup section of this guide lists the versions current as of November 6, 2026.

How does the script decide a link is broken?

It treats a link as broken when the server answers with a status code of 400 or above, or when the request fails entirely, for example with a connection error or a timeout. A 404 or 410 is a real broken link. A 403, 429, or 999 usually means the site blocked the automated request, so check those by hand.

Should I use HEAD or GET to check links?

Use HEAD first, because it fetches headers only and never downloads the body. Fall back to GET when a server answers 405 or 403 to HEAD, since a rejected HEAD request does not always mean a dead page.

How often should I run a broken link check?

Once a week is a sensible default for most sites, which is the schedule Seotal's guide recommends. Run it as a Monday 2 AM cron job if you want it automatic, and run it again right after a redesign or a migration, when links break in bulk.

Is there a CodexFuse tool that checks broken links without code?

Yes. The broken link checker at codexfuse.dev/tools/broken-link-checker runs in your browser with no install and no code. The script in this guide is for readers who want the logic in their own hands and a report they can shape.