Selenium For Web Scraping Python

It is a well-known fact that Python is one of the most popular programming languages for data mining and Web Scraping. There are tons of libraries and niche scrapers around the community, but we’d like to share the 5 most popular of them.

Selenium For Web Scraping Python Example
Selenium For Web Scraping Python Interview

Most of these libraries' advantages can be received by using our API and some of these libraries can be used in stack with it.

The Top 5 Python Web Scraping Libraries in 2020#

Feb 11, 2021 Lean how to scrape the web with Selenium and Python with this step by step tutorial. We will use Selenium to automate Hacker News login. Kevin Sahin Updated: 11 February, 2021 8 min read.

1. Requests#

If you are a beginner to web scraping with Python, check out my guides on Extracting Data from HTML with BeautifulSoup and Crawling the Web with Python and Scrapy. Choose the Right Tool Choosing the right tool depends on the type of project you are working on, since Python has a wide variety of libraries and frameworks for web scraping.
Oct 22, 2015 What is Web Scraping? Web scraping is a computer software technique of extracting information from websites. This technique mostly focuses on the transformation of unstructured data (HTML format) on the web into structured data (database or spreadsheet). You can perform web scraping in various ways, including use of Google Docs to almost every.

Well known library for most of the Python developers as a fundamental tool to get raw HTML data from web resources.

To install the library just execute the following PyPI command in your command prompt or Terminal:

After this you can check installation using REPL:

>>> r = requests.get('https://api.github.com/repos/psf/requests')

'A simple, yet elegant HTTP library.'

Official docs URL: https://requests.readthedocs.io/en/latest/
GitHub repository: https://github.com/psf/requests

2. LXML#

When we’re talking about the speed and parsing of the HTML we should keep in mind this great library called LXML. This is a real champion in HTML and XML parsing while Web Scraping, so the software based on LXML can be used for scraping of frequently-changing pages like gambling sites that provide odds for live events.

To install the library just execute the following PyPI command in your command prompt or Terminal:

The LXML Toolkit is a really powerful instrument and the whole functionality can’t be described in just a few words, so the following links might be very useful:

Official docs URL: https://lxml.de/index.html#documentation
GitHub repository: https://github.com/lxml/lxml/

3. BeautifulSoup#

Probably 80% of all the Python Web Scraping tutorials on the Internet uses the BeautifulSoup4 library as a simple tool for dealing with retrieved HTML in the most human-preferable way. Selectors, attributes, DOM-tree, and much more. The perfect choice for porting code to or from Javascript's Cheerio or jQuery.

To install this library just execute the following PyPI command in your command prompt or Terminal:

As it was mentioned before, there are a bunch of tutorials around the Internet about BeautifulSoup4 usage, so do not hesitate to Google it!

Official docs URL: https://www.crummy.com/software/BeautifulSoup/bs4/doc/
Launchpad repository: https://code.launchpad.net/~leonardr/beautifulsoup/bs4

4. Selenium#

Selenium is the most popular Web Driver that has a lot of wrappers suitable for most programming languages. Quality Assurance engineers, automation specialists, developers, data scientists - all of them at least once used this perfect tool. For the Web Scraping it’s like a Swiss Army knife - there are no additional libraries needed because any action can be performed with a browser like a real user: page opening, button click, form filling, Captcha resolving, and much more.

To install this library just execute the following PyPI command in your command prompt or Terminal:

The code below describes how easy Web Crawling can be started with using Selenium:

Selenium For Web Scraping Python Example

from selenium.webdriver.common.keys import Keys

driver = webdriver.Firefox()

assert'Python'in driver.title

elem.send_keys('pycon')

assert'No results found.'notin driver.page_source

As this example only illustrates 1% of the Selenium power, we’d like to offer of following useful links:

Official docs URL: https://selenium-python.readthedocs.io/
GitHub repository: https://github.com/SeleniumHQ/selenium

5. Scrapy#

Scrapy is the greatest Web Scraping framework, and it was developed by a team with a lot of enterprise scraping experience. The software created on top of this library can be a crawler, scraper, and data extractor or even all this together.

To install this library just execute the following PyPI command in your command prompt or Terminal:

We definitely suggest you start with a tutorial to know more about this piece of gold: https://docs.scrapy.org/en/latest/intro/tutorial.html

As usual, the useful links are below:

Official docs URL: https://docs.scrapy.org/en/latest/index.html
GitHub repository: https://github.com/scrapy/scrapy

What web scraping library to use?#

So, it’s all up to you and up to the task you’re trying to resolve, but always remember to read the Privacy Policy and Terms of the site you’re scraping 😉.

Imagine what would you do if you could automate all the repetitive and boring activities you perform using internet, like checking every day the first results of Google for a given keyword, or download a bunch of files from different websites.

In this post you’ll learn to use Selenium with Python, a Web Scraping tool that simulates a user surfing the Internet. For example, you can use it to automatically look for Google queries and read the results, log in to your social accounts, simulate a user to test your web application, and anything you find in your daily live that it’s repetitive. The possibilities are infinite! 🙂

*All the code in this post has been tested with Python 2.7 and Python 3.4.

Install and use Selenium

Selenium is a python package that can be installed via pip. I recommend that you install it in a virtual environment (using virtualenv and virtualenvwrapper).

To install selenium, you just need to type:

In this post we are going to initialize a Firefox driver — you can install it by visiting their website. However, if you want to work with Chrome or IE, you can find more information here.

Once you have Selenium and Firefox installed, create a python file, selenium_script.py. We are going to initialize a browser using Selenium:

Selenium For Web Scraping Python Interview

</div><table><tbody><tr><td><div><div>2</div><div>4</div><div>6</div><div>8</div><div>10</div><div>12</div><div>14</div><div>16</div><div>18</div><div>20</div><div>22</div><div>24</div><div>26</div><div>28</div><div>30</div><div>32</div></div></td><td><div><div><span>from </span><span>selenium </span><span>import </span><span>webdriver</span></div><div><span>from </span><span>selenium</span><span>.</span><span>webdriver</span><span>.</span><span>support</span><span>.</span><span>ui </span><span>import </span><span>WebDriverWait</span></div><div><span>from </span><span>selenium</span><span>.</span><span>webdriver</span><span>.</span><span>support </span><span>import </span><span>expected_conditions </span><span>as</span><span>EC</span></div><div><span>from </span><span>selenium</span><span>.</span><span>common</span><span>.</span><span>exceptions </span><span>import </span><span>TimeoutException</span></div><div><span>driver</span><span>=</span><span>webdriver</span><span>.</span><span>Firefox</span><span>(</span><span>)</span></div><div><span>return</span><span>driver</span></div><div><span>driver</span><span>.</span><span>get</span><span>(</span><span>'http://www.google.com'</span><span>)</span></div><div><span>box</span><span>=</span><span>driver</span><span>.</span><span>wait</span><span>.</span><span>until</span><span>(</span><span>EC</span><span>.</span><span>presence_of_element_located</span><span>(</span></div><div><span>button</span><span>=</span><span>driver</span><span>.</span><span>wait</span><span>.</span><span>until</span><span>(</span><span>EC</span><span>.</span><span>element_to_be_clickable</span><span>(</span></div><div><span>box</span><span>.</span><span>send_keys</span><span>(</span><span>query</span><span>)</span></div><div><span>except </span><span>TimeoutException</span><span>:</span></div><div><span>if</span><span>__name__</span><span>'__main__'</span><span>:</span></div><div><span>lookup</span><span>(</span><span>driver</span><span>,</span><span>'Selenium'</span><span>)</span></div><div><span>driver</span><span>.</span><span>quit</span><span>(</span><span>)</span></div></div></td></tr></tbody></table><p>In the previous code:</p><ul><li> the function <span>init_driver</span> initializes a driver instance.<ul><li> creates the driver instance</li><li> adds the <span>WebDriverWait</span> function as an attribute to the driver, so it can be accessed more easily. This function is used to make the driver wait a certain amount of time (here 5 seconds) for an event to occur.</li></ul></li><li> the function <span>lookup</span> takes two arguments: a driver instance and a query lookup (a string).<ul><li> it loads the Google search page</li><li> it waits for the query box element to be located and for the button to be clickable. Note that we are using the <span>WebDriverWait</span> function to wait for these elements to appear.</li><li> Both elements are located by name. Other options would be to locate them by <span><span>ID</span><span>,</span><span>XPATH</span><span>,</span><span>TAG_NAME</span><span>,</span><span>CLASS_NAME</span><span>,</span><span>CSS_SELECTOR</span></span> , etc (see table below). You can find more information here.</li><li> Next, it sends the query into the box element and clicks the search button.</li><li> If either the box or button are not located during the time established in the wait function (here, 5 seconds), the <span>TimeoutException</span> is raised.</li></ul></li><li> the next statement is a conditional that is true only when the script is run directly. This prevents the next statements to run when this file is imported.<ul><li> it initializes the driver and calls the lookup function to look for “Selenium”.</li><li> it waits for 5 seconds to see the results and quits the driver</li></ul></li></ul><p>Finally, run your code with:</p><p>Did it work? If you got an <span>ElementNotVisibleException</span> , keep reading!</p><h2>How to catch an ElementNotVisibleExcpetion</h2><p>Google search has recently changed so that initially, Google shows this page:</p><p>and when you start writing your query, the search button moves into the upper part of the screen.</p><p>Well, actually it doesn’t move. The old button becomes invisible and the new one visible (and thus the exception when you click the old one: it’s not visible to click!).</p><p>We can update the lookup function in our code so that it catches this exception:</p><div><textarea wrap='soft' readonly='>from selenium.common.exceptions import ElementNotVisibleException def lookup(driver, query): driver.get('http://www.google.com') try: box = driver.wait.until(EC.presence_of_element_located( (By.NAME, 'q'))) button = driver.wait.until(EC.element_to_be_clickable( (By.NAME, 'btnK'))) box.send_keys(query) try: button.click() except ElementNotVisibleException: button = driver.wait.until(EC.visibility_of_element_located( (By.NAME, 'btnG'))) button.click() except TimeoutException: print('Box or Button not found in google.com')

from selenium.common.exceptions import ElementNotVisibleException

def lookup(driver,query):

try:

box=driver.wait.until(EC.presence_of_element_located(

button=driver.wait.until(EC.element_to_be_clickable(

box.send_keys(query)

button.click()

button=driver.wait.until(EC.visibility_of_element_located(

button.click()

print('Box or Button not found in google.com')

the element that raised the exception, button.click() is inside a try statement.
if the exception is raised, we look for the second button, using visibility_of_element_located to make sure the element is visible, and then click this button.
if at any time, some element is not found within the 5 second period, the TimeoutException is raised and caught by the two end lines of code.
Note that the initial button name is “btnK” and the new one is “btnG”.

Method list in Selenium

To sum up, I’ve created a table with the main methods used here.

Note: it’s not a python file — don’t try to run/import it 🙂

</div><table><tbody><tr><td><div><div>2</div><div>4</div><div>6</div><div>8</div><div>10</div><div>12</div><div>14</div><div>16</div><div>18</div><div>20</div><div>22</div><div>24</div><div>26</div><div>28</div><div>30</div><div>32</div></div></td><td><div><div><span>from </span><span>selenium </span><span>import </span><span>webdriver</span></div><div><span>from </span><span>selenium</span><span>.</span><span>webdriver</span><span>.</span><span>support</span><span>.</span><span>ui </span><span>import </span><span>WebDriverWait</span></div><div><span>driver</span><span>=</span><span>webdriver</span><span>.</span><span>Firefox</span><span>(</span><span>)</span></div><div><span># WAIT FOR ELEMENTS</span></div><div><span>from </span><span>selenium</span><span>.</span><span>webdriver</span><span>.</span><span>support </span><span>import </span><span>expected_conditions </span><span>as</span><span>EC</span></div><div><span>element</span><span>=</span><span>driver</span><span>.</span><span>wait</span><span>.</span><span>until</span><span>(</span></div><div><span>EC</span><span>.</span><span>element_to_be_clickable</span><span>(</span></div><div><span>(</span><span>By</span><span>.</span><span>NAME</span><span>,</span><span>'name'</span><span>)</span></div><div><span>(</span><span>By</span><span>.</span><span>LINK_TEXT</span><span>,</span><span>'link text'</span><span>)</span></div><div><span>(</span><span>By</span><span>.</span><span>TAG_NAME</span><span>,</span><span>'tag name'</span><span>)</span></div><div><span>(</span><span>By</span><span>.</span><span>CSS_SELECTOR</span><span>,</span><span>'css selector'</span><span>)</span></div><div><span>)</span></div><div><span># CATCH EXCEPTIONS</span></div><div><span>TimeoutException</span></div></div></td></tr></tbody></table><p>That’s all! Hope it was useful! 🙂</p><p>Don’t forget to share it with your friends!</p><br><br><a href='https://netlify.mix-goapp.com/adobe-photoshop-6-for-mac-torrent.html#LxzUn=AlVGFwYCD1dVQFFXB1QAAw5fVl4fEVUNXF8MRwtKAlgUT0cAB08VUUVUFQtZAx4UG0MJCV1OAVAIHVUeFhMLUBQFXU4IBQUZBBlJVxlWBVZTG1RKAU5VDxRkNh5XUVMCSApVEQkNAEsZVBUSG0hBARpDEkMAAxVSWGI=' target='_blank'><img src='https://cdn-ak.f.st-hatena.com/images/fotolife/r/ruriatunifoefec/20200910/20200910011354.png' style='cursor:pointer;display:block;margin-left:auto;margin-right:auto;'></a><br><br></p>
<div class="alignright"></div><div class="alignleft"></div> 
                <div class="wp-pagenavi">
                    <div class="alignleft"><a href='/adobe-audition-voice-recording'>Adobe Audition Voice Recording</a></div>                    <div class="alignright"><a href='/sunyou-airmail'>Sunyou Airmail</a></div>                </div>
                                			</div>
			<div id="comments">
	
	
	<p>Comments are closed.</p>
	
</div>
			
		</article>
		
		
		
	</section>
    </div>
<aside id="sidebar" role="complementary">
        
        
    	<div id="archives" class="widget">
			<h3 class="widget-title">MOST POPULAR ARTICLES</h3>
			<ul class="list arrow">
					<li><a href='/chrome-is-out-of-date-mac-red-arrow'>: Chrome Is Out Of Date Mac Red Arrow</a></li>
<li><a href='/microsoft-remote-desktop-macbook'>: Microsoft Remote Desktop Macbook</a></li>
<li><a href='/adobe-acrobat-pro-mac'>: Adobe Acrobat Pro Mac</a></li>
<li><a href='/calibre-mobi'>: Calibre Mobi</a></li>
<li><a href='/silverlight-edge'>: Silverlight Edge</a></li>
<li><a href='/avogadros-number-full'>: Avogadro's Number Full</a></li>
<li><a href='/turbo-350'>: Turbo 350</a></li>
			</ul>
		</div>
		
        
        
</aside>
        </div>
        <footer id="footer">
            <div id="footer-inner" class="row">
                            </div>
        </footer>
        <div id="footer-bar">
            <div class="right">
                <div id="footer-nav">
                                </div>
            </div>
            <div class="left">
                Copyright © 2021 Loadface346. 
	 
		
            </div>
        </div>
    </div>
    
	</body>
</html>