一个神奇的蜘蛛?项目，适用于数据采集任务，源码结构简洁明了。
index页面示例：
—
项目地址https://github.com/lixi5338619/magical_spider
使用指南1、配置settings.py，启动flask服务
2、参考demo文件夹中的代码进行测试，主要通过runflow.py运行。
代码语言：javascript代码运行次数：0运行复制“`javascriptimport requestshost = ‘http://127.0.0.1:5000’def magical_start(project_name,base_url = ‘https://www.php.cn/link/b6f05a7baab2fe0eea07e59bd5b0b317′): # 1、创建浏览器并选择session_id result = requests.post(f'{host}/create’,data={‘name’:project_name,’url’:base_url}).json() session_id,process_url = result[‘session_id’],result[‘process_url’] return session_id,process_urldef magical_request(session_id,process_url,request_url): # 2、请求浏览器_xhr data = {‘session_id’:session_id,’process_url’:process_url, ‘request_url’:request_url,’request_type’:’get’} result = requests.post(f'{host}/xhr’,data=data).json() return result[‘result’]def magical_close(session_id,process_url,process_name): # 4、关闭浏览器 close_data = {‘session_id’:session_id,’process_url’:process_url,’process_name’:process_name} requests.post(f'{host}/close’,data=close_data).json()
3、测试代码GET请求
代码语言：javascript代码运行次数：0<svg fill="none" height="16" viewbox="0 0 16 16" width="16" xmlns="http://www.w3.org/2000/svg"><path d="M6.66666 10.9999L10.6667 7.99992L6.66666 4.99992V10.9999ZM7.99999 1.33325C4.31999 1.33325 1.33333 4.31992 1.33333 7.99992C1.33333 11.6799 4.31999 14.6666 7.99999 14.6666C11.68 14.6666 14.6667 11.6799 14.6667 7.99992C14.6667 4.31992 11.68 1.33325 7.99999 1.33325ZM7.99999 13.3333C5.05999 13.3333 2.66666 10.9399 2.66666 7.99992C2.66666 5.05992 5.05999 2.66659 7.99999 2.66659C10.94 2.66659 13.3333 5.05992 13.3333 7.99992C13.3333 10.9399 10.94 13.3333 7.99999 13.3333Z" fill="currentcolor">运行<svg fill="none" height="16" viewbox="0 0 16 16" width="16" xmlns="http://www.w3.org/2000/svg"><path clip-rule="evenodd" d="M4.5 15.5V3.5H14.5V15.5H4.5ZM12.5 5.5H6.5V13.5H12.5V5.5ZM9.5 2.5H3.5V12.5H1.5V0.5H11.5V2.5H9.5Z" fill="currentcolor" fill-rule="evenodd">复制javascript from demo.runflow import magical_start,magical_request,magical_closeproject_name = 'cnipa'base_url = 'https://www.cnipa.gov.cn'session_id,process_url = magical_start(project_name,base_url)print(len(magical_request(session_id, process_url,'https://www.cnipa.gov.cn/col/col57/index.html')))magical_close(session_id,process_url,project_name)
POST请求
代码语言：javascript代码运行次数：0运行复制javascriptfrom demo.runflow import magical_start,magical_request,magical_closeimport jsonproject_name = 'chinadrugtrials'base_url = 'http://www.chinadrugtrials.org.cn'session_id,process_url = magical_start(project_name,base_url)data = {"id": "","ckm_index": "","sort": "desc","sort2": "","rule": "CTR","secondLevel": "0","currentpage": "2","keywords": "","reg_no": "","indication": "","case_no": "","drugs_name": "","drugs_type": "","appliers": "","communities": "","researchers": "","agencies": "","state": ""}formdata = json.dumps(data)print(magical_request(session_id=session_id, process_url=process_url, request_url='https://www.php.cn/link/44f95c37a24495521a98b63f0bbf4268', request_type='post',formdata=formdata ))magical_close(session_id,process_url,project_name)
4、index页面可以查看和管理当前运行中的任务，同时可以查看系统的内存和磁盘使用情况。
5、demo文件夹包含任务流程汇总runflow.py，以及抖音和药监局的案例，提供了单任务和多任务的示例。
linux部署1.安装chrome（选择安装位置） yum install https://www.php.cn/link/6b17d006a2ed6f12f07c7ea60b8002b5
2.检查chrome版本 google-chrome --version
3.安装对应版本的chromedriver_linux64 例如，我的chrome版本是104.0.5112.79 wget https://www.php.cn/link/5bb5f8d030f5e6e4b6d137c6990f0772
4.解压chromedriver unzip chromedriver_linux64
5.授权chromedriver chmod 777 chromedriver
6.修改项目代码settings.py中的chromedriver路径
7.安装python依赖并启动flask项目
Python依赖：flask、sqlite3、selenium、websockets、opencv-python、numpyflask启动方式：python3 server.py8.开启服务器端口访问权限
9.运行项目测试
以上就是magical_spider远程采集方案的详细内容，更多请关注创想鸟其它相关文章！
版权声明：本文内容由互联网用户自发贡献，该文观点仅代表作者本人。本站仅提供信息存储空间服务，不拥有所有权，不承担相关法律责任。
如发现本站有涉嫌抄袭侵权/违法违规的内容， 请发送邮件至 chuangxiangniao@163.com 举报，一经查实，本站将立刻删除。
发布者：程序猿，转转请注明出处：https://www.chuangxiangniao.com/p/14351.html
微信扫一扫 
支付宝扫一扫 
