Simple Web Scraping Using requests, Beautiful Soup, and lxml in Python

When it comes to web scraping in Python, the first package we think about is Scrapy. However, Scrapy is more suitable for larger scraping projects. Besides, there is a learning curve for Scrapy, which takes time. For simple scraping issues where you only need to get data from a single webpage directly, it can be rather overkilled to use Scrapy. In this case, we can use the requests plus Beautiful Soup/lxml packages to scrape the content you need…

--

--

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store
Lynn Kwong

Lynn Kwong

440 Followers

I’m a Software Developer (https://www.superdataminer.com) keen on sharing thoughts, tutorials, and solutions for the best practice of software development.