What is Web Scraping?

How Web Scraping Works?

Web Scraping with Scrapy

Untitled item

Scrapy Installation for Windows

Scrapy Installation for Ubuntu/Linux

Scrapy Installation for Mac

Scrapy Installation for Anaconda

Creating Scrapy Project

Project Walkthrough

Creating Spider

Sending Request

Getting the Response

Scrapy CSS Selector

Selecting All The Data

Extracting Data

Spider Overview

CSS Selectors v/s XPath : How to Select Web Elements?

Tagname, Class and Id Selectors

Attribute Selectors

XPath Expressions

XPath Attribute Selectors

XPath text( ) Function

What is the Scrapy Shell and How to Use it?

fetch( ) Response

Shell Configuration

Structuring Data Into Scrapy Item

Using Item in Spiders

Define Input and Output Processors For Item Fields

Loading Items with Scrapy ItemLoaders

Items, Processors & ItemLoaders Overview

Output Extracted Data In JSON, CSV & XML Format

Overwrite Previous Output

Appending Data to Previous Output

How to use Scrapy Item Pipelines?

Saving Data Locally to Excel (XLSX) Files

Enable Item Pipelines in Settings

MongoDB (Account) Setup

Saving Data To MonogoDB

Extracting Links From href Attributes

Send Request to the Next Page

start_requests( ) method

How to Follow Links?

How to Select Data Using Regular Expressions With Scrapy

Setting Up Custom Callback Function

Parse Product Details Page

HTML Tables

Selecting Tables Data

Extract Data From HTML Tables

Data Hidden With Logging Forms

Inspecting HTML Forms and Website Activity With Dev Tools

Logging Into Websites With FormRequest

CSRF Protected Login Forms

Extract CSRF Values From Forms

What are JavaScript Rendered/Dynamic Websites?

scrapy-playwright Installation

Setting Up Playwright in Scrapy Project

Using Playwright To Render Websites

Scraping Data From Dynamic Websites

Playwright Overview

Playwright Page Object

Logging In With Playwright

Dynamic Websites With Loading Screens

Wait For Selector/Elements Using Page Couroutines

[Update] PageCoroutine is now PageMethod

Dynamic Websites With Infinite Scroll

Taking Screenshot of Websites

Rendering Websites To PDF

Identifying API Calls

Requesting Data From API

Extracting Data From API

Scrapy Project Settings

Robots.txt File

Middleware

Autothrottle Extension

What are User Agents?

User Agents With Scrapy

What are Proxies?

Proxies With Scrapy

Spider Arguments

Standalone Spiders

Scrapy Shell With bpython

Scrapy get vs extract method

Master Web Scraping and Crawling With Python and Scrapy Framework - In-Depth Guide For Beginners

Andrew So

<p>Web scraping is the process of scraping websites and extracting desired data from the same, and in this course, you&#x27;ll learn and master web scraping using python and scrapy framework with this step-by-step and in-depth guide.</p>
<h3>🪜 A Step-By-Step Guide</h3>
<p>Assuming that you don&#x27;t know anything about web scraping, web crawling, scrapy framework, web scraping, or even web scrapping meaning -  we will <strong>start from the complete basics</strong>. In the first section, you&#x27;ll learn about -</p>
<ul>
<li>Web scraping process step-by-step (with <em>infographics</em> - no code)</li>
<li>How to scrape data from websites and</li>
<li>How to use scrapy for the same (i.e. scrapy meaning)</li>
</ul>
<p>After getting the basics clear and having an idea of how web scraping works, we will start web scraping using python &amp; scrapy framework! Again, we&#x27;ll move step-by-step and perform each step learned in the basics with bite-sized lessons. We&#x27;ll take it slow so that it&#x27;s easier for you to understand every step involved in scraping and extracting data from websites.</p>
<h3>⛏️ Web Scraping &amp; Scrapy Essentials</h3>
<p>Having built an actual web scraper, you&#x27;ll get an idea of how web scraping works firsthand. Now it&#x27;s crucial to cover the <strong>essential concepts of web scraping</strong> and scrapy, which we will do next.</p>
<ul>
<li>CSS Selectors to select web elements</li>
<li>XPath to select web elements</li>
<li>Scrapy Shell to test &amp; verify selectors</li>
<li>Items to organise extracted data</li>
<li>Load Items with ItemLoaders with input &amp; output Processors</li>
<li>Export data to JSON, CSV, XLSX (Excel) &amp; XML file formats</li>
<li>Save extracted data to online databases like MongoDB using ItemPipelines</li>
</ul>
<h3>✨ Master Web Scraping In-Depth</h3>
<p>Learning how to scrape websites and the essentials already makes you a complete web scraper, but we&#x27;ll take this even further and learn the <strong>advanced web scraping techniques</strong> to become an expert!</p>
<ul>
<li>Follow links in a webpage to another page</li>
<li>Crawl multiple pages and extract data, i.e. Pagination</li>
<li>Scrape data using Regular Expressions (RegEx)</li>
<li>Extract Data From HTML Tables</li>
<li>Login Into Websites Using Scrapy FormRequest</li>
<li>Bypass CSRF protected Login forms</li>
<li>Scrape Dynamic or <strong>JavaScript Rendered Websites</strong> using Scrapy Playwright</li>
<li>Interact with web elements like fill forms, and click buttons.</li>
<li>Handle <strong>Infinite Scroll websites</strong></li>
<li>Wait For Elements when contents/data takes time to load</li>
<li>Take Screenshot of websites</li>
<li><strong>Save websites as PDF</strong></li>
<li>Identify API calls from websites and scrape data from APIs</li>
<li>Use middleware in a scrapy project</li>
<li>Use and Rotate User-Agents &amp; Proxies</li>
<li>Web scraping Best Practices</li>
</ul>
<h3>🏗️ Real-World Projects</h3>
<p>After mastering web scraping and web crawling, we need projects to get started! That&#x27;s why you&#x27;ll perform three projects as well:</p>
<ul>
<li>Champions League Table from <strong>ESPN</strong></li>
<li>Product Tracker for <strong>Amazon</strong></li>
<li>Scraper Application with a <strong>GUI</strong></li>
</ul>
<p>Join us in this in-depth course, where you&#x27;ll learn about web scraping from scratch and master the process of extracting data from websites step-by-step. Check out the preview lessons to learn how web scraping works! See you there~</p>

Hi, my name is Rahul Mula and I'm a developer specializing in python, flutter, and web development. I've developed Keyviz - the free and open-source tool to visualize your keystrokes in real-time. I've also authored books and instructed courses on python programming where over a hundred thousand students have joined me so far.

Certificate Details

Web Scraping Masterclass With Scrapy and Python

Course Goals

Rahul Mula