{"id":4520,"date":"2022-03-24T15:46:42","date_gmt":"2022-03-24T07:46:42","guid":{"rendered":"https:\/\/egonlin.com\/?p=4520"},"modified":"2022-04-26T14:14:44","modified_gmt":"2022-04-26T06:14:44","slug":"%e7%ac%ac%e5%85%ad%e7%ab%a0%ef%bc%9ascrapy%e6%a1%86%e6%9e%b6","status":"publish","type":"post","link":"https:\/\/egonlin.com\/?p=4520","title":{"rendered":"\u7b2c\u516d\u7ae0\uff1aScrapy\u6846\u67b6"},"content":{"rendered":"

\u4e00 \u4ecb\u7ecd<\/h1>\n
Scrapy\u4e00\u4e2a\u5f00\u6e90\u548c\u534f\u4f5c\u7684\u6846\u67b6\uff0c\u5176\u6700\u521d\u662f\u4e3a\u4e86\u9875\u9762\u6293\u53d6 (\u66f4\u786e\u5207\u6765\u8bf4, \u7f51\u7edc\u6293\u53d6 )\u6240\u8bbe\u8ba1\u7684\uff0c\u4f7f\u7528\u5b83\u53ef\u4ee5\u4ee5\u5feb\u901f\u3001\u7b80\u5355\u3001\u53ef\u6269\u5c55\u7684\u65b9\u5f0f\u4ece\u7f51\u7ad9\u4e2d\u63d0\u53d6\u6240\u9700\u7684\u6570\u636e\u3002\u4f46\u76ee\u524dScrapy\u7684\u7528\u9014\u5341\u5206\u5e7f\u6cdb\uff0c\u53ef\u7528\u4e8e\u5982\u6570\u636e\u6316\u6398\u3001\u76d1\u6d4b\u548c\u81ea\u52a8\u5316\u6d4b\u8bd5\u7b49\u9886\u57df\uff0c\u4e5f\u53ef\u4ee5\u5e94\u7528\u5728\u83b7\u53d6API\u6240\u8fd4\u56de\u7684\u6570\u636e(\u4f8b\u5982 Amazon Associates Web Services ) \u6216\u8005\u901a\u7528\u7684\u7f51\u7edc\u722c\u866b\u3002\nScrapy \u662f\u57fa\u4e8etwisted\u6846\u67b6\u5f00\u53d1\u800c\u6765\uff0ctwisted\u662f\u4e00\u4e2a\u6d41\u884c\u7684\u4e8b\u4ef6\u9a71\u52a8\u7684python\u7f51\u7edc\u6846\u67b6\u3002\u56e0\u6b64Scrapy\u4f7f\u7528\u4e86\u4e00\u79cd\u975e\u963b\u585e\uff08\u53c8\u540d\u5f02\u6b65\uff09\u7684\u4ee3\u7801\u6765\u5b9e\u73b0\u5e76\u53d1\u3002\u6574\u4f53\u67b6\u6784\u5927\u81f4\u5982\u4e0b<\/code><\/pre>\n
\"\"<\/div><\/p>\n
The data flow in Scrapy is controlled by the execution engine, and goes like this:<\/p>\n
The Engine gets the initial Requests to crawl from the Spider.
\nThe Engine schedules the Requests in the Scheduler and asks for the next Requests to crawl.
\nThe Scheduler returns the next Requests to the Engine.
\nThe Engine sends the Requests to the Downloader, passing through the Downloader Middlewares (see process_request()).
\nOnce the page finishes downloading the Downloader generates a Response (with that page) and sends it to the Engine, passing through the Downloader Middlewares (see process_response()).
\nThe Engine receives the Response from the Downloader and sends it to the Spider for processing, passing through the Spider Middleware (see process_spider_input()).
\nThe Spider processes the Response and returns scraped items and new Requests (to follow) to the Engine, passing through the Spider Middleware (see process_spider_output()).
\nThe Engine sends processed items to Item Pipelines, then send processed Requests to the Scheduler and asks for possible next Requests to crawl.
\nThe process repeats (from step 1) until there are no more requests from the Scheduler.<\/p>\n
Components\uff1a<\/p>\n
\u5f15\u64ce(EGINE)
\n\u5f15\u64ce\u8d1f\u8d23\u63a7\u5236\u7cfb\u7edf\u6240\u6709\u7ec4\u4ef6\u4e4b\u95f4\u7684\u6570\u636e\u6d41\uff0c\u5e76\u5728\u67d0\u4e9b\u52a8\u4f5c\u53d1\u751f\u65f6\u89e6\u53d1\u4e8b\u4ef6\u3002\u6709\u5173\u8be6\u7ec6\u4fe1\u606f\uff0c\u8bf7\u53c2\u89c1\u4e0a\u9762\u7684\u6570\u636e\u6d41\u90e8\u5206\u3002<\/p>\n
\u8c03\u5ea6\u5668(SCHEDULER)
\n\u7528\u6765\u63a5\u53d7\u5f15\u64ce\u53d1\u8fc7\u6765\u7684\u8bf7\u6c42, \u538b\u5165\u961f\u5217\u4e2d, \u5e76\u5728\u5f15\u64ce\u518d\u6b21\u8bf7\u6c42\u7684\u65f6\u5019\u8fd4\u56de. \u53ef\u4ee5\u60f3\u50cf\u6210\u4e00\u4e2aURL\u7684\u4f18\u5148\u7ea7\u961f\u5217, \u7531\u5b83\u6765\u51b3\u5b9a\u4e0b\u4e00\u4e2a\u8981\u6293\u53d6\u7684\u7f51\u5740\u662f\u4ec0\u4e48, \u540c\u65f6\u53bb\u9664\u91cd\u590d\u7684\u7f51\u5740
\n\u4e0b\u8f7d\u5668(DOWLOADER)
\n\u7528\u4e8e\u4e0b\u8f7d\u7f51\u9875\u5185\u5bb9, \u5e76\u5c06\u7f51\u9875\u5185\u5bb9\u8fd4\u56de\u7ed9EGINE\uff0c\u4e0b\u8f7d\u5668\u662f\u5efa\u7acb\u5728twisted\u8fd9\u4e2a\u9ad8\u6548\u7684\u5f02\u6b65\u6a21\u578b\u4e0a\u7684
\n\u722c\u866b(SPIDERS)
\nSPIDERS\u662f\u5f00\u53d1\u4eba\u5458\u81ea\u5b9a\u4e49\u7684\u7c7b\uff0c\u7528\u6765\u89e3\u6790responses\uff0c\u5e76\u4e14\u63d0\u53d6items\uff0c\u6216\u8005\u53d1\u9001\u65b0\u7684\u8bf7\u6c42
\n\u9879\u76ee\u7ba1\u9053(ITEM PIPLINES)
\n\u5728items\u88ab\u63d0\u53d6\u540e\u8d1f\u8d23\u5904\u7406\u5b83\u4eec\uff0c\u4e3b\u8981\u5305\u62ec\u6e05\u7406\u3001\u9a8c\u8bc1\u3001\u6301\u4e45\u5316\uff08\u6bd4\u5982\u5b58\u5230\u6570\u636e\u5e93\uff09\u7b49\u64cd\u4f5c
\n\u4e0b\u8f7d\u5668\u4e2d\u95f4\u4ef6(Downloader Middlewares)
\n\u4f4d\u4e8eScrapy\u5f15\u64ce\u548c\u4e0b\u8f7d\u5668\u4e4b\u95f4\uff0c\u4e3b\u8981\u7528\u6765\u5904\u7406\u4eceEGINE\u4f20\u5230DOWLOADER\u7684\u8bf7\u6c42request\uff0c\u5df2\u7ecf\u4eceDOWNLOADER\u4f20\u5230EGINE\u7684\u54cd\u5e94response\uff0c\u4f60\u53ef\u7528\u8be5\u4e2d\u95f4\u4ef6\u505a\u4ee5\u4e0b\u51e0\u4ef6\u4e8b
\nprocess a request just before it is sent to the Downloader (i.e. right before Scrapy sends the request to the website);
\nchange received response before passing it to a spider;
\nsend a new Request instead of passing received response to a spider;
\npass response to a spider without fetching a web page;
\nsilently drop some requests.
\n\u722c\u866b\u4e2d\u95f4\u4ef6(Spider Middlewares)
\n\u4f4d\u4e8eEGINE\u548cSPIDERS\u4e4b\u95f4\uff0c\u4e3b\u8981\u5de5\u4f5c\u662f\u5904\u7406SPIDERS\u7684\u8f93\u5165\uff08\u5373responses\uff09\u548c\u8f93\u51fa\uff08\u5373requests\uff09
\n\u5b98\u7f51\u94fe\u63a5\uff1ahttps:\/\/docs.scrapy.org\/en\/latest\/topics\/architecture.html<\/a><\/p>\n
\u4e8c \u5b89\u88c5<\/h1>\n
Windows\u5e73\u53f0<\/h2>\n
1\u3001pip3 install wheel #\u5b89\u88c5\u540e\uff0c\u4fbf\u652f\u6301\u901a\u8fc7wheel\u6587\u4ef6\u5b89\u88c5\u8f6f\u4ef6\uff0cwheel\u6587\u4ef6\u5b98\u7f51\uff1ahttps:\/\/www.lfd.uci.edu\/~gohlke\/pythonlibs\n3\u3001pip3 install lxml\n4\u3001pip3 install pyopenssl\n5\u3001\u4e0b\u8f7d\u5e76\u5b89\u88c5pywin32\uff1ahttps:\/\/sourceforge.net\/projects\/pywin32\/files\/pywin32\/\n6\u3001\u4e0b\u8f7dtwisted\u7684wheel\u6587\u4ef6\uff1ahttp:\/\/www.lfd.uci.edu\/~gohlke\/pythonlibs\/#twisted\n7\u3001\u6267\u884cpip3 install \u4e0b\u8f7d\u76ee\u5f55\\Twisted-17.9.0-cp36-cp36m-win_amd64.whl\n8\u3001pip3 install scrapy<\/code><\/pre>\n
Linux\u5e73\u53f0<\/h2>\n
1\u3001pip3 install scrapy<\/code><\/pre>\n
\u4e09 \u547d\u4ee4\u884c\u5de5\u5177<\/h1>\n
#1 \u67e5\u770b\u5e2e\u52a9\nscrapy -h\nscrapy  -h\n\n#2 \u6709\u4e24\u79cd\u547d\u4ee4\uff1a\u5176\u4e2dProject-only\u5fc5\u987b\u5207\u5230\u9879\u76ee\u6587\u4ef6\u5939\u4e0b\u624d\u80fd\u6267\u884c\uff0c\u800cGlobal\u7684\u547d\u4ee4\u5219\u4e0d\u9700\u8981\nGlobal commands:\n    startproject #\u521b\u5efa\u9879\u76ee\n    genspider    #\u521b\u5efa\u722c\u866b\u7a0b\u5e8f\n    settings     #\u5982\u679c\u662f\u5728\u9879\u76ee\u76ee\u5f55\u4e0b\uff0c\u5219\u5f97\u5230\u7684\u662f\u8be5\u9879\u76ee\u7684\u914d\u7f6e\n    runspider    #\u8fd0\u884c\u4e00\u4e2a\u72ec\u7acb\u7684python\u6587\u4ef6\uff0c\u4e0d\u5fc5\u521b\u5efa\u9879\u76ee\n    shell        #scrapy shell url\u5730\u5740  \u5728\u4ea4\u4e92\u5f0f\u8c03\u8bd5\uff0c\u5982\u9009\u62e9\u5668\u89c4\u5219\u6b63\u786e\u4e0e\u5426\n    fetch        #\u72ec\u7acb\u4e8e\u7a0b\u5355\u7eaf\u5730\u722c\u53d6\u4e00\u4e2a\u9875\u9762\uff0c\u53ef\u4ee5\u62ff\u5230\u8bf7\u6c42\u5934\n    view         #\u4e0b\u8f7d\u5b8c\u6bd5\u540e\u76f4\u63a5\u5f39\u51fa\u6d4f\u89c8\u5668\uff0c\u4ee5\u6b64\u53ef\u4ee5\u5206\u8fa8\u51fa\u54ea\u4e9b\u6570\u636e\u662fajax\u8bf7\u6c42\n    version      #scrapy version \u67e5\u770bscrapy\u7684\u7248\u672c\uff0cscrapy version -v\u67e5\u770bscrapy\u4f9d\u8d56\u5e93\u7684\u7248\u672c\nProject-only commands:\n    crawl        #\u8fd0\u884c\u722c\u866b\uff0c\u5fc5\u987b\u521b\u5efa\u9879\u76ee\u624d\u884c\uff0c\u786e\u4fdd\u914d\u7f6e\u6587\u4ef6\u4e2dROBOTSTXT_OBEY = False\n    check        #\u68c0\u6d4b\u9879\u76ee\u4e2d\u6709\u65e0\u8bed\u6cd5\u9519\u8bef\n    list         #\u5217\u51fa\u9879\u76ee\u4e2d\u6240\u5305\u542b\u7684\u722c\u866b\u540d\n    edit         #\u7f16\u8f91\u5668\uff0c\u4e00\u822c\u4e0d\u7528\n    parse        #scrapy parse url\u5730\u5740 --callback \u56de\u8c03\u51fd\u6570  #\u4ee5\u6b64\u53ef\u4ee5\u9a8c\u8bc1\u6211\u4eec\u7684\u56de\u8c03\u51fd\u6570\u662f\u5426\u6b63\u786e\n    bench        #scrapy bentch\u538b\u529b\u6d4b\u8bd5\n\n#3 \u5b98\u7f51\u94fe\u63a5\nhttps:\/\/docs.scrapy.org\/en\/latest\/topics\/commands.html<\/code><\/pre>\n
1 \u67e5\u770b\u5e2e\u52a9<\/h2>\n
scrapy -h\nscrapy  -h<\/code><\/pre>\n
2 \u6709\u4e24\u79cd\u547d\u4ee4\uff1a\u5176\u4e2dProject-only\u5fc5\u987b\u5207\u5230\u9879\u76ee\u6587\u4ef6\u5939\u4e0b\u624d\u80fd\u6267\u884c\uff0c\u800cGlobal\u7684\u547d\u4ee4\u5219\u4e0d\u9700\u8981<\/h2>\n
Global commands:\n    startproject #\u521b\u5efa\u9879\u76ee\n    genspider    #\u521b\u5efa\u722c\u866b\u7a0b\u5e8f\n    settings     #\u5982\u679c\u662f\u5728\u9879\u76ee\u76ee\u5f55\u4e0b\uff0c\u5219\u5f97\u5230\u7684\u662f\u8be5\u9879\u76ee\u7684\u914d\u7f6e\n    runspider    #\u8fd0\u884c\u4e00\u4e2a\u72ec\u7acb\u7684python\u6587\u4ef6\uff0c\u4e0d\u5fc5\u521b\u5efa\u9879\u76ee\n    shell        #scrapy shell url\u5730\u5740  \u5728\u4ea4\u4e92\u5f0f\u8c03\u8bd5\uff0c\u5982\u9009\u62e9\u5668\u89c4\u5219\u6b63\u786e\u4e0e\u5426\n    fetch        #\u72ec\u7acb\u4e8e\u7a0b\u5355\u7eaf\u5730\u722c\u53d6\u4e00\u4e2a\u9875\u9762\uff0c\u53ef\u4ee5\u62ff\u5230\u8bf7\u6c42\u5934\n    view         #\u4e0b\u8f7d\u5b8c\u6bd5\u540e\u76f4\u63a5\u5f39\u51fa\u6d4f\u89c8\u5668\uff0c\u4ee5\u6b64\u53ef\u4ee5\u5206\u8fa8\u51fa\u54ea\u4e9b\u6570\u636e\u662fajax\u8bf7\u6c42\n    version      #scrapy version \u67e5\u770bscrapy\u7684\u7248\u672c\uff0cscrapy version -v\u67e5\u770bscrapy\u4f9d\u8d56\u5e93\u7684\u7248\u672c\nProject-only commands:\n    crawl        #\u8fd0\u884c\u722c\u866b\uff0c\u5fc5\u987b\u521b\u5efa\u9879\u76ee\u624d\u884c\uff0c\u786e\u4fdd\u914d\u7f6e\u6587\u4ef6\u4e2dROBOTSTXT_OBEY = False\n    check        #\u68c0\u6d4b\u9879\u76ee\u4e2d\u6709\u65e0\u8bed\u6cd5\u9519\u8bef\n    list         #\u5217\u51fa\u9879\u76ee\u4e2d\u6240\u5305\u542b\u7684\u722c\u866b\u540d\n    edit         #\u7f16\u8f91\u5668\uff0c\u4e00\u822c\u4e0d\u7528\n    parse        #scrapy parse url\u5730\u5740 --callback \u56de\u8c03\u51fd\u6570  #\u4ee5\u6b64\u53ef\u4ee5\u9a8c\u8bc1\u6211\u4eec\u7684\u56de\u8c03\u51fd\u6570\u662f\u5426\u6b63\u786e\n    bench        #scrapy bentch\u538b\u529b\u6d4b\u8bd5\n\n## 3 \u5b98\u7f51\u94fe\u63a5\nhttps:\/\/docs.scrapy.org\/en\/latest\/topics\/commands.html<\/code><\/pre>\n
\u793a\u8303\u7528\u6cd5<\/p>\n
1\u3001\u6267\u884c\u5168\u5c40\u547d\u4ee4\uff1a\u8bf7\u786e\u4fdd\u4e0d\u5728\u67d0\u4e2a\u9879\u76ee\u7684\u76ee\u5f55\u4e0b\uff0c\u6392\u9664\u53d7\u8be5\u9879\u76ee\u914d\u7f6e\u7684\u5f71\u54cd<\/h1>\n
scrapy startproject MyProject\n\ncd MyProject\nscrapy genspider baidu www.baidu.com\n\nscrapy settings --get XXX #\u5982\u679c\u5207\u6362\u5230\u9879\u76ee\u76ee\u5f55\u4e0b\uff0c\u770b\u5230\u7684\u5219\u662f\u8be5\u9879\u76ee\u7684\u914d\u7f6e\n\nscrapy runspider baidu.py\n\nscrapy shell https:\/\/www.baidu.com\n    response\n    response.status\n    response.body\n    view(response)\n\nscrapy view https:\/\/www.taobao.com #\u5982\u679c\u9875\u9762\u663e\u793a\u5185\u5bb9\u4e0d\u5168\uff0c\u4e0d\u5168\u7684\u5185\u5bb9\u5219\u662fajax\u8bf7\u6c42\u5b9e\u73b0\u7684\uff0c\u4ee5\u6b64\u5feb\u901f\u5b9a\u4f4d\u95ee\u9898\n\nscrapy fetch --nolog --headers https:\/\/www.taobao.com\n\nscrapy version #scrapy\u7684\u7248\u672c\n\nscrapy version -v #\u4f9d\u8d56\u5e93\u7684\u7248\u672c`\n\n#2\u3001\u6267\u884c\u9879\u76ee\u547d\u4ee4\uff1a\u5207\u5230\u9879\u76ee\u76ee\u5f55\u4e0b\nscrapy crawl baidu\nscrapy check\nscrapy list\nscrapy parse http:\/\/quotes.toscrape.com\/ --callback parse\nscrapy bench<\/code><\/pre>\n
\u56db \u9879\u76ee\u7ed3\u6784\u4ee5\u53ca\u722c\u866b\u5e94\u7528\u7b80\u4ecb<\/h1>\n
project_name\/
\nscrapy.cfg
\nproject_name\/
\ninit<\/strong>.py
\nitems.py
\npipelines.py
\nsettings.py
\nspiders\/
\ninit<\/strong>.py
\n\u722c\u866b1.py
\n\u722c\u866b2.py
\n\u722c\u866b3.py
\n\u6587\u4ef6\u8bf4\u660e\uff1a<\/p>\n
scrapy.cfg  \u9879\u76ee\u7684\u4e3b\u914d\u7f6e\u4fe1\u606f\uff0c\u7528\u6765\u90e8\u7f72scrapy\u65f6\u4f7f\u7528\uff0c\u722c\u866b\u76f8\u5173\u7684\u914d\u7f6e\u4fe1\u606f\u5728settings.py\u6587\u4ef6\u4e2d\u3002
\nitems.py    \u8bbe\u7f6e\u6570\u636e\u5b58\u50a8\u6a21\u677f\uff0c\u7528\u4e8e\u7ed3\u6784\u5316\u6570\u636e\uff0c\u5982\uff1aDjango\u7684Model
\npipelines    \u6570\u636e\u5904\u7406\u884c\u4e3a\uff0c\u5982\uff1a\u4e00\u822c\u7ed3\u6784\u5316\u7684\u6570\u636e\u6301\u4e45\u5316
\nsettings.py \u914d\u7f6e\u6587\u4ef6\uff0c\u5982\uff1a\u9012\u5f52\u7684\u5c42\u6570\u3001\u5e76\u53d1\u6570\uff0c\u5ef6\u8fdf\u4e0b\u8f7d\u7b49\u3002\u5f3a\u8c03:\u914d\u7f6e\u6587\u4ef6\u7684\u9009\u9879\u5fc5\u987b\u5927\u5199\u5426\u5219\u89c6\u4e3a\u65e0\u6548\uff0c\u6b63\u786e\u5199\u6cd5USER_AGENT=’xxxx’
\nspiders      \u722c\u866b\u76ee\u5f55\uff0c\u5982\uff1a\u521b\u5efa\u6587\u4ef6\uff0c\u7f16\u5199\u722c\u866b\u89c4\u5219
\n\u6ce8\u610f\uff1a\u4e00\u822c\u521b\u5efa\u722c\u866b\u6587\u4ef6\u65f6\uff0c\u4ee5\u7f51\u7ad9\u57df\u540d\u547d\u540d<\/p>\n
\u9ed8\u8ba4\u53ea\u80fd\u5728cmd\u4e2d\u6267\u884c\u722c\u866b\uff0c\u5982\u679c\u60f3\u5728pycharm\u4e2d\u6267\u884c\u9700\u8981\u505a
\n\u5173\u4e8ewindows\u7f16\u7801<\/p>\n
\u4e94 Spiders<\/h1>\n
<\/div>
\u67e5\u770b\u66f4\u591a<\/h2>
\u8054\u7cfb\u7ba1\u7406\u5458\u5fae\u4fe1tutu19192010\uff0c\u6ce8\u518c\u8d26\u53f7<\/p>\n<\/div>
<\/div>
<\/td><\/tr>
<\/td><\/tr><\/tbody><\/table>
<\/td><\/tr><\/tbody><\/table><\/form>
Register<\/a><\/div>
Forgotten username or password?<\/a><\/div>
<\/div><\/div><\/div>
<\/div>","protected":false},"excerpt":{"rendered":"
\u4e00 \u4ecb\u7ecd Scrapy\u4e00\u4e2a\u5f00\u6e90\u548c\u534f\u4f5c\u7684\u6846\u67b6\uff0c\u5176\u6700\u521d\u662f\u4e3a\u4e86\u9875\u9762\u6293\u53d6 (\u66f4\u786e\u5207\u6765\u8bf4, \u7f51\u7edc\u6293\u53d6 )\u6240\u8bbe\u8ba1\u7684\uff0c\u4f7f\u7528\u5b83 […]<\/p>\n","protected":false},"author":3,"featured_media":0,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":[],"categories":[342,345],"tags":[],"_links":{"self":[{"href":"https:\/\/egonlin.com\/index.php?rest_route=\/wp\/v2\/posts\/4520"}],"collection":[{"href":"https:\/\/egonlin.com\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/egonlin.com\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/egonlin.com\/index.php?rest_route=\/wp\/v2\/users\/3"}],"replies":[{"embeddable":true,"href":"https:\/\/egonlin.com\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=4520"}],"version-history":[{"count":0,"href":"https:\/\/egonlin.com\/index.php?rest_route=\/wp\/v2\/posts\/4520\/revisions"}],"wp:attachment":[{"href":"https:\/\/egonlin.com\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=4520"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/egonlin.com\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=4520"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/egonlin.com\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=4520"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}