A python Scrapy tutorial

A python Scrapy tutorial

https://www.youtube.com/watch?v=OJ8isyws2yw

Python scrapy is a web crawling package.

The youtuber gave a scrapy example.

First type scrapy startproject tutorial

You will get a directory, tutorial, under tutorial/spiders, generate a python file called quotes_spider.py under spiders directory.



The code:

import scrapy

class QuotesSpider(scrapy.Spider):
   name="quotes"
   def start_requests(self):
       urls=[
          'http://quotes.toscrape.com/page/1/',
          'http://quotes.toscrape.com/page/2/'
           ]
       for url in urls:
           yield scrapy.Request(url=url,callback=self.parse)
   def parse(self, response):
        page=response.url.split("/")[-2]
        filename="quotes-%s.html"%page
        with open(filename,"wb") as f:
                f.write(response.body)
       self.log("Saved file %s" % filename)



   

   



How to run it:
under spiders directory run
scrapy crawl quotes





Results, in spiders folder:

got two quotes-.html files.