枫枫知道个人博客|Scrapy爬虫框架

[[ slide_text ]]

[[ item.content ]]

Scrapy爬虫框架

发布时间：2022-10-25 作者：枫枫来源：枫枫知道个人博客

python 爬虫

# 安装与创建项目

```Vue
pip install scrapy
```

创建项目

```Vue
scrapy startproject demo
```

```Vue
scrapy genspider douban movie.douban.com
```

运行项目

```Vue
scrapy crawl douban -o douban.csv

```

优化导入

```Vue
ctrl alt o
```

![](http://python.fengfengzhidao.com/pic/20220814212558.png)

# 管道

## 使用管道将数据写入Excel文件

pipelines.py

```Python
from itemadapter import ItemAdapter
import openpyxl

class DemoPipeline:

def __init__(self):
        self.wb = openpyxl.Workbook()
        # wb.create_sheet()
        self.sheet = self.wb.active
        self.sheet.title = 'Top250'
        self.sheet.append(('标题', '评分', '评分人数', '封面'))
    
    # 打开爬虫的钩子函数
    def open_spider(self, spider):
        pass 
    
    # 关闭爬虫的钩子函数
    def close_spider(self, spider):
        self.wb.save('电影数据.xlsx')
    
    # 处理数据
    def process_item(self, item, spider):
        title = item.get('title', '')
        star = item.get('star', '')
        people = item.get('people', '')
        cover = item.get('cover', '')
        self.sheet.append((
            title,
            star,
            people,
            cover,
        ))
        return item
```

配置管道

```Python
ITEM_PIPELINES = {
   'demo.pipelines.DemoPipeline': 300,
}
```

## 写入mysql

```SQL
create database spider;
use spider;
drop table if exists tb_top_movie;
create table tb_top_movie (
    mov_id int unsigned auto_increment comment '编号',
    title varchar(50) not null comment '标题',
    star decimal(3,1) not null comment '评分',
    people varchar(16) not null comment '人数',
    cover varchar(400) not null comment '封面',
    primary key (mov_id)
) engine=innodb comment='Top电源表';

```

```Python
class DBPipeline:

def __init__(self):
        print('db管道执行了')
        self.conn = pymysql.connect(
            host='127.0.0.1',
            port=3306,
            user='root',
            password='root',
            database='spider',
            charset='utf8mb4'
        )
        self.cursor = self.conn.cursor()

def close_spider(self, spider):
        # 提交
        self.conn.commit()
        # 关闭连接
        self.conn.close()

def process_item(self, item, spider):
        title = item.get('title', '')
        star = item.get('star', 0)
        people = item.get('people', '')
        cover = item.get('cover', '')

self.cursor.execute(
            'insert into tb_top_movie (title, star, people, cover) values (%s, %s, %s, %s)',
            (title, star, people, cover)
        )

return item

```

批处理

```Python
class DBPipeline:

def close_spider(self, spider):
        if len(self.data) > 0:
            self._write_to_db()

# 关闭连接
        self.conn.close()

# 写入数据库
    def _write_to_db(self):
        self.cursor.executemany(
            'insert into tb_top_movie (title, star, people, cover) values (%s, %s, %s, %s)',
            self.data
        )
        self.conn.commit()

def process_item(self, item, spider):
        title = item.get('title', '')
        star = item.get('star', 0)
        people = item.get('people', '')
        cover = item.get('cover', '')
        self.data.append((title, star, people, cover))
        if len(self.data) == 100:
            self._write_to_db()
            self.data.clear()
        return item
```

# 中间件

# 从更多页面解析数据

```Python
class DoubanSpider(scrapy.Spider):
    name = 'douban'
    allowed_domains = ['movie.douban.com']

def start_requests(self):
        for page in range(1):
            yield Request(
                url=f'https://movie.douban.com/top250?start={page * 25}&filter=',
                # meta={'proxy': 'sock5://127.0.0.1:1086'},
                # callback=self.parse  # 默认就是这个
            )

def parse(self, response: HtmlResponse, *args, **kwargs):
        sel = Selector(response)
        list_items = sel.css('#content > div > div.article > ol > li > div.item')
        for list_item in list_items:
            detail_url = list_item.css('.info > div.hd > a::attr(href)').extract_first()
            movie_item = MovieItem()
            movie_item['title'] = list_item.css('.info .title::text').extract_first()
            movie_item['star'] = list_item.css('.info .star .rating_num::text').extract_first()
            movie_item['people'] = list_item.css('.info .star span:nth-child(4)::text').extract_first()
            movie_item['cover'] = list_item.css('.pic img::attr(src)').extract_first()
            yield Request(
                url=detail_url,
                callback=self.detail_parse,  # 回调的解析方法
                cb_kwargs={"item": movie_item}  # 将movie_item传递给解析方法
            )

# 解析电影的详情
    def detail_parse(self, response: HtmlResponse, *args, **kwargs):
        movie_item = kwargs['item']
        sel = Selector(response)
        movie_item['duration'] = sel.css('span[property="v:summary"]::text').extract_first()
        movie_item['intro'] = sel.css('span[property="v:runtime"]::text').extract_first()
        yield movie_item
```

上一篇：rabbitMQ消息队列下一篇：web框架之flask

你觉得文章怎么样

Iwtbk

![emoji](https://face.t.sinajs.cn/t4/appstyle/expression/ext/normal/c9/2018new_chongjing_org.png)

7个月前
点赞（0）回复（0）

枫枫知道个人博客

Scrapy爬虫框架

Iwtbk