CodeCrafting Explained

A Comprehensive Guide to Text Extraction in Selenium with Python

J. V. G. Ransika's photo
J. V. G. Ransika
·

3 min read

Table of contents

Today we are going to look at one of the common tasks in web automation, extracting text from web elements. In this article, we will explore three different methods for text extraction in Selenium using Python.

To demonstrate the text extraction methods, we'll use a simple HTML page. Here's the HTML code:

<!DOCTYPE html>
<html>
<head>
    <title>Selenium Text Extraction Example</title>
</head>
<body>
    <div id="element1">This is the visible text of element 1.</div>
    <div id="element2" style="display: none;">This is hidden text.</div>
    <div id="element3">This is the visible text of element 3, and <span>some <strong>inner text</strong></span>.</div>
</body>
</html>

This HTML page contains three different elements with various text content and visibility, making it ideal for showcasing our text extraction methods.

Method 1: get_attribute("textContent")

This method allows us to extract the visible text content of an element. Here's a Python code snippet demonstrating this:

from selenium import webdriver

# Initialize the Chrome web driver
driver = webdriver.Chrome()

# Open the HTML page in the browser
driver.get("file:///path/to/your/example.html")

# Extract text using get_attribute("textContent")
element1 = driver.find_element_by_id("element1")
text_content_1 = element1.get_attribute("textContent")
print("Method 1 - get_attribute('textContent'):", text_content_1)

# Close the browser
driver.quit()

Output :

Method 1 - get_attribute('textContent'): This is the visible text of element 1.

Method 2: get_attribute("innerHTML")

This method returns the HTML content within the element, including any HTML tags. Here's the code:

# ...
# Initialize the Chrome web driver and open the HTML page (same as before)

# Extract HTML content using get_attribute("innerHTML")
element3 = driver.find_element_by_id("element3")
inner_html_3 = element3.get_attribute("innerHTML")
print("Method 2 - get_attribute('innerHTML'):", inner_html_3)

# Close the browser
driver.quit()

Output :

Method 2 - get_attribute('innerHTML'): This is the visible text of element 3, and <span>some <strong>inner text</strong></span>.

Method 3: Using CSS Selectors and text Attribute

The third method involves using CSS selectors and the text attribute to extract visible text content. Here's the code:

# ...
# Initialize the Chrome web driver and open the HTML page (same as before)

# Extract visible text using CSS selector and text attribute
element3 = driver.find_element_by_id("element3")
visible_text_3 = element3.text
print("Method 3 - CSS selector and text attribute:", visible_text_3)

# Close the browser
driver.quit()

Output :

Method 3 - CSS selector and text attribute: This is the visible text of element 3, and some inner text.

Comparison and Conclusion

Let's summarize when to use each method :

  • Use get_attribute("textContent") when you want the visible text content of an element, including hidden or non-displayed text.

  • Use get_attribute("innerHTML") when you need the entire content, including HTML tags.

  • Use CSS selectors with the text attribute when you want just the visible text.

In this article, we explored three different methods for text extraction in Selenium using Python. You should now have a clear understanding of how to use these methods and when to choose one over the other. Selenium is a versatile tool for web automation, and knowing how to extract text is an essential skill for web testing and scraping.

If you found this article helpful, please share it with others. Feel free to leave comments and questions below, and don't forget to subscribe for more tutorials and tips on web automation and programming.

Thank you for reading!

Pythonseleniumtext extractionPython 3python beginner