Here is how to make a webpage request in Python.


# Import the requests library
import requests
  
# Visit a page with Python
response = requests.get('https://example.com')

# Get the content of visited page
content = response.content
print(content) 

Output

The output is, of course, the source code of the webpage. In our case, the printed source code would look something like this:

b'<!doctype html>\n<html>\n<head>\n    <title>Example Domain</title>\n\n    <meta charset="utf-8" />\n    <meta http-equiv="Content-type" content="text/html; charset=utf-8" />\n    <meta name="viewport" content="width=device-width, initial-scale=1" />\n    <style type="text/css">\n    body {\n        background-color: #f0f0f2;\n        margin: 0;\n        padding: 0;\n        font-family: -apple-system, system-ui, BlinkMacSystemFont, "Segoe UI", "Open Sans", "Helvetica Neue", Helvetica, Arial, sans-serif;\n        \n    }\n    div {\n        width: 600px;\n        margin: 5em auto;\n        padding: 2em;\n        background-color: #fdfdff;\n        border-radius: 0.5em;\n        box-shadow: 2px 3px 7px 2px rgba(0,0,0,0.02);\n    }\n    a:link, a:visited {\n        color: #38488f;\n        text-decoration: none;\n    }\n    @media (max-width: 700px) {\n        div {\n            margin: 0 auto;\n            width: auto;\n        }\n    }\n    </style>    \n</head>\n\n<body>\n<div>\n    <h1>Example Domain</h1>\n    <p>This domain is for use in illustrative examples in documents. You may use this\n    domain in literature without prior coordination or asking for permission.</p>\n    <p><a href="https://www.iana.org/domains/example">More information...</a></p>\n</div>\n</body>\n</html>\n'


Explanation

Note: You need to install the Python requests library with pip install requests

The Python code above makes an HTTP request to the given URL just like browsers do. The server that hosts the webpage returns a response object which contains the content of that webpage. Python will extract the content of the webpage using the content property. The content is actually the webpage source code and inside that source code there are the data which you may want to scrape. To scrape the data you need to add additional code that extracts the data from the source code. A good library to scrape data with Python is BeautifulSoup.


Related HowTos
Do dictionary comprehension
Do list comprehension
Create a virtual environment
Create a new file
Merge two lists
Extract items from two different lists into one list
Check if a text file is empty
Randomly select an item from a list
Generate a random integer
Break a while loop
Create a pandas DataFrame from a dictionary
Create a pandas DataFrame from a list
Get the last item of a list
Delete a column from a pandas dataframe
Access a column of a pandas dataframe
Create a class
Get the first two characters of a string
<django.db.models.fields.CharField>
Recommended Course

Learn Flask development and learn to build cool apps with our premium Python course on Udemy.


Mailing List

We will keep you Python-motivated if you subscribe below.