Here is how to use Selenium to browse a page in Python.


1. Install Selenium and WebDriver Manager

Let's start by installing the necessary libraries using pip:

pip install selenium webdriver-manager

2. Writing the Script

Create a Python script (e.g., selenium_wikipedia.py) and open it in your favorite code editor.

3. Example Script: Opening Wikipedia and Clicking a Link

Here is a complete example script that opens Wikipedia, searches for a term, and clicks on the first search result link:

from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.common.keys import Keys
from selenium.webdriver.chrome.service import Service
from webdriver_manager.chrome import ChromeDriverManager
import time # Set up the Chrome WebDriver
service = Service(ChromeDriverManager().install())
driver = webdriver.Chrome(service=service) try:
# Open the Wikipedia homepage
driver.get("https://www.wikipedia.org/") # Locate the search input field by its name attribute and enter a search term
search_input = driver.find_element(By.NAME, 'search')
search_input.send_keys('Selenium (software)') # Submit the search form
search_input.send_keys(Keys.RETURN) # Wait for the results page to load and display the results
time.sleep(2) # Click on the first search result link
first_result = driver.find_element(By.CSS_SELECTOR, 'ul.mw-search-results li a')
first_result.click() # Optional: Wait for a few seconds to ensure the page loads completely
time.sleep(5) # Print the title of the current page
print("Page title after click:", driver.title) finally:
# Close the browser
driver.quit()

4. Running the Script

Run the script using your terminal or command prompt:

python selenium_wikipedia.py

5. Explanation of the Script

  1. Setting Up WebDriver:

    • The script uses the webdriver_manager library to automatically manage the Chrome WebDriver.
    • It sets up the Chrome WebDriver using webdriver.Chrome(service=service).
  2. Opening Wikipedia:

    • The script opens the Wikipedia homepage using driver.get("https://www.wikipedia.org/").
  3. Searching for a Term:

    • The script locates the search input field by its name attribute using driver.find_element(By.NAME, 'search').
    • It enters the search term 'Selenium (software)' and submits the search form using search_input.send_keys(Keys.RETURN).
  4. Clicking on the First Search Result:

    • The script waits for the search results page to load using time.sleep(2).
    • It locates the first search result link using driver.find_element(By.CSS_SELECTOR, 'ul.mw-search-results li a') and clicks on it.
  5. Printing the Page Title:

    • The script waits for a few seconds to ensure the page loads completely using time.sleep(5).
    • It prints the title of the current page to verify that the click was successful using print("Page title after click:", driver.title).
  6. Closing the Browser:

    • The script closes the browser using driver.quit().

Output

The script in the example starts the Chrome browser, visits the Wikipedia.org page, searches for "Selenium (software)" in the Wikipedia search box, and clicks on the link returned by the search results.


Explanation

 



Related HowTos
Use the polars library
Load JSON data
Build a GUI with FreeSimpleGUI
Delete a directory
Delete a file
Create matplotlib graphs
Rename a file
Get today's date and convert to string
Create a text file and write text on it
Read a text file with Python
Scrape a Wikipedia page
Install dependencies
Flush the output of the print function
Prettyprint a JSON file
Create a dictionary with list comprehension
Select multiple columns in a Pandas dataframe
Profile a script
Reverse a string
Convert two lists into a dictionary
Convert an integer into a string
Generate random strings with upper case letters and digits
Extract extension from filename
Print to stderr in Python
Generate random integers between 0 and 9
Use a pythonic way to create a long multi-line string
Measure elapsed time in Python
Install specific package versions with pip
Read from stdin (standard input)
Get the class name of an instance
Check if a string is empty
Pad zeroes to a string
Delete an element from a dictionary
Check if a string is a float
Count the occurrences of an item in a list
Remove an element from a list by index
Use static methods
Remove a trailing newline
Print literal curly-brace characters in a string and also use .format on it
Exit/deactivate a virtualenv
Determine the type of an object
Limit floats to two decimal points
Know if an object has an attribute
Select an item from a list randomly
Read a file line-by-line into a list
Call a function of a module by using its name
Get the number of elements in a list
Print without a newline or space
Sort a list of dictionaries by a value of the dictionary
Remove a key from dictionary
Rename column names with Pandas
Lowercase a string
Upgrade all Python packages with pip
Get a substring of a string
Get the last element of a list
Parse a string to a float or integer
Convert a string into datetime format
Access environment variable values
Print coloured text on the terminal
Find current directory of a file
Change the size of figures drawn with Matplotlib
Manually raise an exception
Delete a file or folder
Split a list into evenly sized parts
Select rows from a DataFrame based on column values with Pandas
Install pip on Windows
Check if a given key already exists in a dictionary
Iterate over rows in a DataFrame for Pandas
Make function decorators and chain them together
Pass a variable by reference
Make a time delay
Convert bytes to a string
Copy a file
Concatenate two lists
Add new keys to a dictionary
Catch multiple exceptions in a single 'except' block
Check if a list is empty
Get the current time
Sort a dictionary by value
Use global variables in a function
List all files of a directory
Iterate over dictionaries using for loops
Check if a string contains a substring
Find the index of an item in a list
Understand how slice notation works
Make a flat list out of a list of lists
Accesses the index in for loops
Safely create a nested directory
Merge two dictionaries in a single expression
Check whether a file exists without exceptions
Convert a dictionary into a list
Convert a list into a dictionary
Duplicate a file
Append text in a text file
Use for loops
Deploy a web app to Heroku
Schedule a Python script for execution at a specific time every day
Store Python passwords securely on Windows, Mac, and Linux
Do dictionary comprehension
Do list comprehension
Create a virtual environment
Create a new file
Merge two lists
Extract items from two different lists into one list
Check if a text file is empty
Randomly select an item from a list
Generate a random integer
Break a while loop
Create a pandas DataFrame from a dictionary
Create a pandas DataFrame from a list
Get the last item of a list
Delete a column from a pandas dataframe
Access a column of a pandas dataframe
Create a class
Make a webpage request
Get the first two characters of a string
Loop through two lists at the same time
Recommended Course

Python Mega Course: Learn Python in 60 Days, Build 20 Apps
Learn Python on Udemy completely in 60 days or less by building 20 real-world applications from web development to data science.