网络爬虫（又被称为网页蜘蛛，网络机器人，在FOAF社区中间，更经常的称为网页追逐者），是一种按照一定的规则，自动地抓取万维网信息的程序或者脚本。另外一些不常使用的名字还有蚂蚁、自动索引、模拟程序或者蠕虫。
Python标准库中提供了：urllib、urllib2、httplib等模块以供Http请求，但是，它的 API 太渣了。它是为另一个时代、另一个互联网所创建的。它需要巨量的工作，甚至包括各种方法覆盖，来完成最简单的任务。
Requests 是使用 Apache2 Licensed 许可证的 基于Python开发的HTTP 库，其在Python内置模块的基础上进行了高度的封装，从而使得Pythoner进行网络请求时，变得美好了许多，使用Requests可以轻而易举的完成浏览器可有的任何操作。
1、GET请求
| 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 | # 1、无参数实例 import requests ret = requests.get( 'https://github.com/timeline.json' ) print ret.url print ret.text # 2、有参数实例 import requests payload = { 'key1' : 'value1' , 'key2' : 'value2' } ret = requests.get( "http://httpbin.org/get" , params = payload) print ret.url print ret.text |
向 https://github.com/timeline.json 发送一个GET请求，将请求和响应相关均封装在 ret 对象中。
2、POST请求
| 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 | # 1、基本POST实例 import requests payload = { 'key1' : 'value1' , 'key2' : 'value2' } ret = requests.post( "http://httpbin.org/post" , data = payload) print ret.text # 2、发送请求头和数据实例 import requests import json url = 'https://api.github.com/some/endpoint' payload = { 'some' : 'data' } headers = { 'content-type' : 'application/json' } ret = requests.post(url, data = json.dumps(payload), headers = headers) print ret.text print ret.cookies |
向https://api.github.com/some/endpoint发送一个POST请求，将请求和相应相关的内容封装在 ret 对象中。
3、其他请求
| 1 2 3 4 5 6 7 8 9 10 | requests.get(url, params = None , * * kwargs) requests.post(url, data = None , json = None , * * kwargs) requests.put(url, data = None , * * kwargs) requests.head(url, * * kwargs) requests.delete(url, * * kwargs) requests.patch(url, data = None , * * kwargs) requests.options(url, * * kwargs) # 以上方法均是在此方法的基础上构建 requests.request(method, url, * * kwargs) |
requests模块已经将常用的Http请求方法为用户封装完成，用户直接调用其提供的相应方法即可
官方文档：http://cn.python-requests.org/zh_CN/latest/user/quickstart.html#id4
BeautifulSoup
BeautifulSoup是一个模块，该模块用于接收一个HTML或XML字符串，然后将其进行格式化，之后遍可以使用他提供的方法进行快速查找指定元素，从而使得在HTML或XML中查找指定元素变得简单。
| 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 | from bs4 import BeautifulSoup html_doc = """ <html><head><title>The Dormouse's story</title></head> <body> asdf <div class="title"> <b>The Dormouse's story总共</b> <h1>f</h1> </div> <div class="story">Once upon a time there were three little sisters; and their names were <a class="sister0" id="link1">Els<span>f</span>ie</a>, <a href="http://example.com/lacie" class="sister" id="link2">Lacie</a> and <a href="http://example.com/tillie" class="sister" id="link3">Tillie</a>; and they lived at the bottom of a well.</div> ad<br/>sf <p class="story">...</p> </body> </html> """ soup = BeautifulSoup(html_doc, features = "lxml" ) # 找到第一个a标签 tag1 = soup.find(name = 'a' ) # 找到所有的a标签 tag2 = soup.find_all(name = 'a' ) # 找到id＝link2的标签 tag3 = soup.select( '#link2' ) |
安装：
| 1 | pip3 install beautifulsoup4 |
使用示例：
| 1 2 3 4 5 6 7 8 9 10 11 | from bs4 import BeautifulSoup html_doc = """ <html><head><title>The Dormouse's story</title></head> <body> ... </body> </html> """ soup = BeautifulSoup(html_doc, features = "lxml" ) |
1. name，标签名称
| 1 2 3 4 5 | # tag = soup.find('a') # name = tag.name # 获取 # print(name) # tag.name = 'span' # 设置 # print(soup) |
2. attr，标签属性
| 1 2 3 4 5 6 | # tag = soup.find('a') # attrs = tag.attrs # 获取 # print(attrs) # tag.attrs = {'ik':123} # 设置 # tag.attrs['id'] = 'iiiii' # 设置 # print(soup) |
3. children,所有子标签
| 1 2 | # body = soup.find('body') # v = body.children |
4. children,所有子子孙孙标签
| 1 2 | # body = soup.find('body') # v = body.descendants |
5. clear,将标签的所有子标签全部清空（保留标签名）
| 1 2 3 | # tag = soup.find('body') # tag.clear() # print(soup) |
6. decompose,递归的删除所有的标签
| 1 2 3 | # body = soup.find('body') # body.decompose() # print(soup) |
7. extract,递归的删除所有的标签，并获取删除的标签
| 1 2 3 | # body = soup.find('body') # v = body.extract() # print(soup) |
8. decode,转换为字符串（含当前标签）；decode_contents（不含当前标签）
| 1 2 3 4 | # body = soup.find('body') # v = body.decode() # v = body.decode_contents() # print(v) |
9. encode,转换为字节（含当前标签）；encode_contents（不含当前标签）
| 1 2 3 4 | # body = soup.find('body') # v = body.encode() # v = body.encode_contents() # print(v) |
10. find,获取匹配的第一个标签
| 1 2 3 4 5 | # tag = soup.find('a') # print(tag) # tag = soup.find(name='a', attrs={'class': 'sister'}, recursive=True, text='Lacie') # tag = soup.find(name='a', class_='sister', recursive=True, text='Lacie') # print(tag) |
11. find_all,获取匹配的所有标签
| 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 | # tags = soup.find_all('a') # print(tags) # tags = soup.find_all('a',limit=1) # print(tags) # tags = soup.find_all(name='a', attrs={'class': 'sister'}, recursive=True, text='Lacie') # # tags = soup.find(name='a', class_='sister', recursive=True, text='Lacie') # print(tags) # ####### 列表 ####### # v = soup.find_all(name=['a','div']) # print(v) # v = soup.find_all(class_=['sister0', 'sister']) # print(v) # v = soup.find_all(text=['Tillie']) # print(v, type(v[0])) # v = soup.find_all(id=['link1','link2']) # print(v) # v = soup.find_all(href=['link1','link2']) # print(v) # ####### 正则 ####### import re # rep = re.compile('p') # rep = re.compile('^p') # v = soup.find_all(name=rep) # print(v) # rep = re.compile('sister.*') # v = soup.find_all(class_=rep) # print(v) # rep = re.compile('http://www.oldboy.com/static/.*') # v = soup.find_all(href=rep) # print(v) # ####### 方法筛选 ####### # def func(tag): # return tag.has_attr('class') and tag.has_attr('id') # v = soup.find_all(name=func) # print(v) # ## get,获取标签属性 # tag = soup.find('a') # v = tag.get('id') # print(v) |
12. has_attr,检查标签是否具有该属性
| 1 2 3 | # tag = soup.find('a') # v = tag.has_attr('id') # print(v) |
13. get_text,获取标签内部文本内容
| 1 2 3 | # tag = soup.find('a') # v = tag.get_text('id') # print(v) |
14. index,检查标签在某标签中的索引位置
| 1 2 3 4 5 6 7 | # tag = soup.find('body') # v = tag.index(tag.find('div')) # print(v) # tag = soup.find('body') # for i,v in enumerate(tag): # print(i,v) |
15. is_empty_element,是否是空标签(是否可以是空)或者自闭合标签，
判断是否是如下标签：'br' , 'hr', 'input', 'img', 'meta','spacer', 'link', 'frame', 'base'
| 1 2 3 | # tag = soup.find('br') # v = tag.is_empty_element # print(v) |
16. 当前的关联标签
| 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 | # soup.next # soup.next_element # soup.next_elements # soup.next_sibling # soup.next_siblings # # tag.previous # tag.previous_element # tag.previous_elements # tag.previous_sibling # tag.previous_siblings # # tag.parent # tag.parents |
17. 查找某标签的关联标签
| 1 2 3 4 5 6 7 8 9 10 11 12 13 14 | # tag.find_next(...) # tag.find_all_next(...) # tag.find_next_sibling(...) # tag.find_next_siblings(...) # tag.find_previous(...) # tag.find_all_previous(...) # tag.find_previous_sibling(...) # tag.find_previous_siblings(...) # tag.find_parent(...) # tag.find_parents(...) # 参数同find_all |
18. select,select_one, CSS选择器
| 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 | soup.select( "title" ) soup.select( "p nth-of-type(3)" ) soup.select( "body a" ) soup.select( "html head title" ) tag = soup.select( "span,a" ) soup.select( "head > title" ) soup.select( "p > a" ) soup.select( "p > a:nth-of-type(2)" ) soup.select( "p > #link1" ) soup.select( "body > a" ) soup.select( "#link1 ~ .sister" ) soup.select( "#link1 + .sister" ) soup.select( ".sister" ) soup.select( "[class~=sister]" ) soup.select( "#link1" ) soup.select( "a#link2" ) soup.select( 'a[href]' ) soup.select( 'a[href="http://example.com/elsie"]' ) soup.select( 'a[href^="http://example.com/"]' ) soup.select( 'a[href$="tillie"]' ) soup.select( 'a[href*=".com/el"]' ) from bs4.element import Tag def default_candidate_generator(tag): for child in tag.descendants: if not isinstance (child, Tag): continue if not child.has_attr( 'href' ): continue yield child tags = soup.find( 'body' ).select( "a" , _candidate_generator = default_candidate_generator) print ( type (tags), tags) from bs4.element import Tag def default_candidate_generator(tag): for child in tag.descendants: if not isinstance (child, Tag): continue if not child.has_attr( 'href' ): continue yield child tags = soup.find( 'body' ).select( "a" , _candidate_generator = default_candidate_generator, limit = 1 ) print ( type (tags), tags) |
19. 标签的内容
| 1 2 3 4 5 6 7 8 9 10 11 12 13 | # tag = soup.find('span') # print(tag.string) # 获取 # tag.string = 'new content' # 设置 # print(soup) # tag = soup.find('body') # print(tag.string) # tag.string = 'xxx' # print(soup) # tag = soup.find('body') # v = tag.stripped_strings # 递归内部获取所有标签的文本 # print(v) |
20.append在当前标签内部追加一个标签
| 1 2 3 4 5 6 7 8 9 10 | # tag = soup.find('body') # tag.append(soup.find('a')) # print(soup) # # from bs4.element import Tag # obj = Tag(name='i',attrs={'id': 'it'}) # obj.string = '我是一个新来的' # tag = soup.find('body') # tag.append(obj) # print(soup) |
21.insert在当前标签内部指定位置插入一个标签
| 1 2 3 4 5 6 | # from bs4.element import Tag # obj = Tag(name='i', attrs={'id': 'it'}) # obj.string = '我是一个新来的' # tag = soup.find('body') # tag.insert(2, obj) # print(soup) |
22. insert_after,insert_before 在当前标签后面或前面插入
| 1 2 3 4 5 6 7 | # from bs4.element import Tag # obj = Tag(name='i', attrs={'id': 'it'}) # obj.string = '我是一个新来的' # tag = soup.find('body') # # tag.insert_before(obj) # tag.insert_after(obj) # print(soup) |
23. replace_with 在当前标签替换为指定标签
| 1 2 3 4 5 6 | # from bs4.element import Tag # obj = Tag(name='i', attrs={'id': 'it'}) # obj.string = '我是一个新来的' # tag = soup.find('div') # tag.replace_with(obj) # print(soup) |
24. 创建标签之间的关系
| 1 2 3 4 | # tag = soup.find('div') # a = soup.find('a') # tag.setup(previous_sibling=a) # print(tag.previous_sibling) |
25. wrap，将指定标签把当前标签包裹起来
| 1 2 3 4 5 6 7 8 9 10 11 | # from bs4.element import Tag # obj1 = Tag(name='div', attrs={'id': 'it'}) # obj1.string = '我是一个新来的' # # tag = soup.find('a') # v = tag.wrap(obj1) # print(soup) # tag = soup.find('a') # v = tag.wrap(soup.find('p')) # print(soup) |
26. unwrap，去掉当前标签，将保留其包裹的标签
| 1 2 3 | # tag = soup.find('a') # v = tag.unwrap() # print(soup) |
更多参数官方：http://beautifulsoup.readthedocs.io/zh_CN/v4.4.0/