Python中怎么定时执行网站爬虫
Python中怎么定时执行网站爬虫，针对这个问题，这篇文章详细介绍了相对应的分析和解答，希望可以帮助更多想解决这个问题的小伙伴找到更简单易行的方法。
成都创新互联专注于盘龙企业网站建设,响应式网站开发,商城系统网站开发。盘龙网站建设公司,为盘龙等地区提供建站服务。全流程专业公司，专业设计，全程项目跟踪，成都创新互联专业和态度为您提供的服务
编写爬虫代码
编写一个爬虫程序，使用requests与beautifulsoup4包爬取和解析Yahoo!股市-上市成交价排行与Yahoo!股市-上柜成交价排行的资料，再利用pandas包将解析后的展示出来。
import datetime
import requests
from bs4 import BeautifulSoup
import pandas as pd
def get_price_ranks():
current_dt = datetime.datetime.now().strftime("%Y-%m-%d %X")
current_dts = [current_dt for _ in range(200)]
stock_types = ["tse", "otc"]
price_rank_urls = ["https://tw.stock.yahoo.com/d/i/rank.php?t=pri&e={}&n=100".format(st) for st in stock_types]
tickers = []
stocks = []
prices = []
volumes = []
mkt_values = []
ttl_steps = 10*100
each_step = 10
for pr_url in price_rank_urls:
r = requests.get(pr_url)
soup = BeautifulSoup(r.text, 'html.parser')
ticker = [i.text.split()[0] for i in soup.select(".name a")]
tickers += ticker
stock = [i.text.split()[1] for i in soup.select(".name a")]
stocks += stock
price = [float(soup.find_all("td")[2].find_all("td")[i].text) for i in range(5, 5+ttl_steps, each_step)]
prices += price
volume = [int(soup.find_all("td")[2].find_all("td")[i].text.replace(",", "")) for i in range(11, 11+ttl_steps, each_step)]
volumes += volume
mkt_value = [float(soup.find_all("td")[2].find_all("td")[i].text)*100000000 for i in range(12, 12+ttl_steps, each_step)]
mkt_values += mkt_value
types = ["上市" for _ in range(100)] + ["上柜" for _ in range(100)]
ky_registered = [True if "KY" in st else False for st in stocks]
df = pd.DataFrame()
df["scrapingTime"] = current_dts
df["type"] = types
df["kyRegistered"] = ky_registered
df["ticker"] = tickers
df["stock"] = stocks
df["price"] = prices
df["volume"] = volumes
df["mktValue"] = mkt_values
return df
price_ranks = get_price_ranks()
print(price_ranks.shape)这个的结果展示为
## (200, 8)
接下来我们利用pandas进行前几行展示
price_ranks.head()
price_ranks.tail()
接下来我们就开始往服务器上部署
对于服务器的选择，环境配置不在本课的讨论范围之内，我们主要是要讲一下怎么去设置定时任务。
接下来我们改造一下代码，改造成结果有sqlite存储。
import datetime
import requests
from bs4 import BeautifulSoup
import pandas as pd
import sqlite3
def get_price_ranks():
current_dt = datetime.datetime.now().strftime("%Y-%m-%d %X")
current_dts = [current_dt for _ in range(200)]
stock_types = ["tse", "otc"]
price_rank_urls = ["https://tw.stock.yahoo.com/d/i/rank.php?t=pri&e={}&n=100".format(st) for st in stock_types]
tickers = []
stocks = []
prices = []
volumes = []
mkt_values = []
ttl_steps = 10*100
each_step = 10
for pr_url in price_rank_urls:
r = requests.get(pr_url)
soup = BeautifulSoup(r.text, 'html.parser')
ticker = [i.text.split()[0] for i in soup.select(".name a")]
tickers += ticker
stock = [i.text.split()[1] for i in soup.select(".name a")]
stocks += stock
price = [float(soup.find_all("td")[2].find_all("td")[i].text) for i in range(5, 5+ttl_steps, each_step)]
prices += price
volume = [int(soup.find_all("td")[2].find_all("td")[i].text.replace(",", "")) for i in range(11, 11+ttl_steps, each_step)]
volumes += volume
mkt_value = [float(soup.find_all("td")[2].find_all("td")[i].text)*100000000 for i in range(12, 12+ttl_steps, each_step)]
mkt_values += mkt_value
types = ["上市" for _ in range(100)] + ["上櫃" for _ in range(100)]
ky_registered = [True if "KY" in st else False for st in stocks]
df = pd.DataFrame()
df["scrapingTime"] = current_dts
df["type"] = types
df["kyRegistered"] = ky_registered
df["ticker"] = tickers
df["stock"] = stocks
df["price"] = prices
df["volume"] = volumes
df["mktValue"] = mkt_values
return df
price_ranks = get_price_ranks()
conn = sqlite3.connect('/home/ubuntu/yahoo_stock.db')
price_ranks.to_sql("price_ranks", conn, if_exists="append", index=False)接下来如果我们让他定时启动，那么，我们需要linux的crontab命令：
如果我们要设置每天的 9:30 到 16:30 之间每小时都执行一次
那么我们只需要先把文件命名为price_rank_scraper.py
关于Python中怎么定时执行网站爬虫问题的解答就分享到这里了，希望以上内容可以对大家有一定的帮助，如果你还有很多疑惑没有解开，可以关注创新互联行业资讯频道了解更多相关知识。
名称栏目：Python中怎么定时执行网站爬虫
新闻来源：http://mswzjz.cn/article/jisogd.html