Sutra Zhou
|
0e01550a79
|
Upgrade dependencies, including the jedis from 2.9.3 to 3.4.1.
|
2021-01-06 03:21:10 +08:00 |
|
Sutra Zhou
|
5d14efc50f
|
Serialize request URL only in FileCacheQueueScheduler.
|
2020-06-14 00:20:39 +08:00 |
|
Sutra Zhou
|
b98a87e45a
|
Serialize requests in FileCacheQueueScheduler, so that the extra info of request could be restored.
|
2020-04-11 20:21:20 +08:00 |
yihua.huang
|
c701fe8d38
|
#702 Refactor: rename CheckForAdditionalInfo to checkForAdditionalInfo
|
2017-11-30 11:50:52 +08:00 |
yihy
|
266083fa07
|
[Fix] #698 Repair using redis,Request additional information is lost
|
2017-11-29 20:19:00 +08:00 |
|
yihua.huang
|
b879b0eed0
|
fix redisscheduler #583
|
2017-06-01 22:25:01 +08:00 |
|
yihua.huang
|
d87c73b472
|
change check-and-set to atomic sadd for redis DuplicateRemover #368
|
2017-03-01 22:24:34 +08:00 |
|
yihua.huang
|
00dfebbceb
|
#424 remove guava dep and add fix docs
|
2016-12-18 10:45:50 +08:00 |
|
yihua.huang
|
a960a39c44
|
fix compile error for example change
|
2016-12-18 08:32:14 +08:00 |
|
yihua.huang
|
b92e6b04f0
|
#400 修复FileCacheQueueScheduler自己设置DuplicateRemover会导致NPE的问题
|
2016-11-25 08:30:24 +08:00 |
Salon.sai
|
f89a6a6826
|
add: redis scheduler with priority
|
2016-07-05 16:29:01 +08:00 |
|
Yihua Huang
|
cfde3b7657
|
Merge pull request #237 from SpenceZhou/master
Update RedisScheduler.java
|
2015-12-02 22:17:00 +08:00 |
SpenceZhou
|
165e5a72eb
|
Update RedisScheduler.java
修改redisscheduler中获取爬取总数bug
|
2015-12-02 17:10:42 +08:00 |
x1ny
|
90e14b31b0
|
修正FileCacheQueueScheduler导致程序不能正常结束和未关闭流
FileCacheQueueScheduler中开启了一个线程周期运行来保存数据但在爬虫结束后没有关闭导致程序无法结束，以及没有关闭io流。
解决方法：
让FileCacheQueueScheduler实现Closable接口，在close方法中关闭线程以及流。
在Spider的close方法中添加对scheduler的关闭操作。
|
2015-11-12 23:10:20 +08:00 |
|
yihua.huang
|
8551b668a0
|
remove commented code
|
2014-09-29 14:51:36 +08:00 |
zhugw
|
eb3c78b9d8
|
Update FileCacheQueueScheduler.java
这样是不是更严谨? 否则的话,中断后再次启动时, (第一个)入口地址仍会被添加到队列及写入到文件中.
但是现在有另外一个问题存在,如第一遍全部抓取完毕了(通过spider.getStatus==Stopped判断),休眠24小时,再来抓取(通过递归调用抓取方法).
这时不同于中断后再启动,lineReader==cursor, 于是初始化时队列为空,入口地址又在urls集合中了, 故导致抓取线程马上就结束了.这样的话就没有办法去抓取网站上的新增内容了.
解决方案一:
判断抓取完毕后,紧接着覆盖cursor文件,第二次来抓取时,curosr为0, 于是将urls.txt中的所有url均放入队列中了, 可以通过这些url来发现新增url.
方案二:
对方案一进行优化,方案一虽然可以满足业务要求,但会做很多无用功,如仍会对所有旧target url进行下载,抽取,持久化等操作.而新增的内容一般都会在HelpUrl中, 比如某一页多了一个新帖子,或者多了几页内容. 故第二遍及以后来爬取时可以仅将HelpUrl放入队列中.
希望能给予反馈,我上述理解对不对, 有什么没有考虑到的情况或者有更简单的方案?谢谢!
|
2014-09-14 16:20:03 +08:00 |
|
yihua.huang
|
42a30074c9
|
update urls.contains to DuplicateRemover in FileCacheQueueScheduler #157
|
2014-09-12 07:52:38 +08:00 |
|
zhugw
|
1db940a088
|
Update FileCacheQueueScheduler.java
在使用过程中发现urls.txt文件存在重复URL的情况,经跟踪源代码,发现初始化加载文件后,读取所有的url放入一集合中,但是之后添加待抓取URL时并未判断是否已存在该集合中(即文件中)了,故导致文件中重复URL的情况.故据此对源码做了修改,还请作者审阅.
|
2014-09-11 15:46:09 +08:00 |
|
yihua.huang
|
186b90512e
|
refactor redisscheduler #118
|
2014-05-02 20:24:15 +08:00 |
|
yihua.huang
|
d1140b9e29
|
add bloom filter for scheduler #118
|
2014-05-02 20:20:22 +08:00 |
|
yihua.huang
|
e8d4a9be2b
|
fix remove duplicate error #117
|
2014-04-29 20:32:06 +08:00 |
|
yihua.huang
|
1104122979
|
more abstraction in scheduler
|
2014-04-27 09:30:01 +08:00 |
|
yihua.huang
|
11ba5beb42
|
[refactor]move monitor to webmagic-extension #98
|
2014-04-25 13:17:13 +08:00 |
|
yihua.huang
|
a5db6cf292
|
some monitor and JMX support #98
|
2014-04-17 00:35:09 +08:00 |
|
yihua.huang
|
c1e7207869
|
add FileCacheQueueScheduler support for cycleRetryTimes
|
2014-04-07 11:00:09 +08:00 |
|
yihua.huang
|
969ad1766b
|
change logger style to slf4j for cleaner code
|
2014-04-06 21:32:20 +08:00 |
|
yihua.huang
|
0e98183f74
|
Change log4j to slf4j #55
|
2014-02-12 09:35:57 +08:00 |
jon
|
a722f9bb66
|
修复由于FileCacheQueueScheduler中fileCursor 文件再次打开时没有初始化抛出NullPointerException的错误
|
2014-01-08 21:24:58 +08:00 |
|
yihua.huang
|
b838c4e433
|
#34 Close reader in FileCacheQueueScheduler
|
2013-11-08 14:59:09 +08:00 |
|
yihua.huang
|
d141541ef3
|
add retry
|
2013-09-04 09:57:19 +08:00 |
|
yihua.huang
|
c70ed57025
|
remove PriorityScheduler to core
|
2013-08-20 21:55:58 +08:00 |
|
yihua.huang
|
7003426898
|
update pom
|
2013-08-20 21:52:39 +08:00 |
|
yihua.huang
|
c79d6ecf09
|
complete all comments
|
2013-08-17 23:30:49 +08:00 |
|
yihua.huang
|
5f1f4cbc46
|
update comments
|
2013-08-17 20:41:29 +08:00 |
|
yihua.huang
|
59aad6a7f4
|
comments in english
|
2013-08-17 18:33:05 +08:00 |
|
yihua.huang
|
f21097421b
|
add new constructor to redisscheduler
|
2013-08-11 18:53:13 +08:00 |
|
yihua.huang
|
0f2c5b5723
|
update redisscheduler
|
2013-08-11 18:28:12 +08:00 |
|
yihua.huang
|
21eca688e9
|
complete docs
|
2013-08-09 20:56:33 +08:00 |
|
yihua.huang
|
17d2d98cec
|
remove invalid @date
|
2013-08-09 20:43:06 +08:00 |
|
yihua.huang
|
cff943f698
|
fix path format error
|
2013-08-07 13:05:12 +08:00 |
|
yihua.huang
|
0c8599e3b2
|
update packages structure
|
2013-08-06 23:17:07 +08:00 |