How to download forum pages to HTML using Python & requests

This will download page 1000 … page 1 in descending order and save the HTML to 1000.html up to 1.html

download_forum_pages.py

import requests
for i in range(1000, 0, -1):
    with open(f"{i}.html", "w") as outfile:
        outfile.write(requests.get(f"https://forum.my-domain.com/showthread.php?123456-my-thread/page{i}").text)

import requests
for i in range(1000, 0, -1):
    with open(f"{i}.html", "w") as outfile:
        outfile.write(requests.get(f"https://forum.my-domain.com/showthread.php?123456-my-thread/page{i}").text)

With progress bar

download_forum_pages_with_progress.py

import requests
from tqdm import tqdm
for i in tqdm(range(1000, 0, -1)):
    with open(f"{i}.html", "w") as outfile:
        outfile.write(requests.get(f"https://forum.my-domain.com/showthread.php?123456-my-thread/page{i}").text)

import requests
from tqdm import tqdm
for i in tqdm(range(1000, 0, -1)):
    with open(f"{i}.html", "w") as outfile:
        outfile.write(requests.get(f"https://forum.my-domain.com/showthread.php?123456-my-thread/page{i}").text)

Check out similar posts by category: Python

If this post helped you, please consider buying me a coffee or donating via PayPal to support research & publishing of new posts on TechOverflow

Buy me a coffee