yihua.huang
|
76766a7c77
|
RequestUtils for range #222
|
2017-06-05 17:13:33 +08:00 |
|
yihua.huang
|
5de6205563
|
use RawText as source for JsonPath #589
|
2017-06-04 07:24:56 +08:00 |
|
yihua.huang
|
211ac74389
|
use group 0 of targetUrl instead of group 1 #588
|
2017-06-04 07:11:13 +08:00 |
|
yihua.huang
|
9b77306098
|
test of ExtractByUrl #586
|
2017-06-04 07:03:20 +08:00 |
|
yihua.huang
|
d8bd0637a1
|
add custom formatter test #586
|
2017-06-04 06:51:41 +08:00 |
|
yihua.huang
|
6cc647577a
|
fix test for ExtractLinks #586
|
2017-06-03 22:38:42 +08:00 |
|
yihua.huang
|
a6f8ed5476
|
complete formatter refactor by ObjectFormatterBuilder #586
|
2017-06-03 22:07:53 +08:00 |
|
yihua.huang
|
b1ef61b278
|
add tests before refactor #586
|
2017-06-03 21:33:18 +08:00 |
|
yihua.huang
|
b363ee6a9d
|
move constant to class
|
2017-06-03 20:33:28 +08:00 |
|
yihua.huang
|
df682857a7
|
fix test and some refactor
|
2017-06-03 20:19:41 +08:00 |
|
yihua.huang
|
0e6eb46eba
|
add test for jsonpath in pagemodel #462
|
2017-06-03 16:47:21 +08:00 |
|
yihua.huang
|
f5018d569e
|
remove test_issue409 #409
|
2017-06-03 15:10:25 +08:00 |
|
yihua.huang
|
17ae500c77
|
test_issue409 #409
|
2017-06-03 15:09:51 +08:00 |
|
yihua.huang
|
4cd5b4f93e
|
test_issue409
|
2017-06-03 15:02:33 +08:00 |
|
yihua.huang
|
fb0acd710c
|
complete SimpleHttpClientTest
|
2017-06-03 14:59:08 +08:00 |
|
yihua.huang
|
d07941d900
|
SimpleHttpClientTest
|
2017-06-03 14:58:34 +08:00 |
|
yihua.huang
|
f02f469c69
|
add test #570
|
2017-06-03 11:02:31 +08:00 |
|
yihua.huang
|
2d693580fc
|
add test
|
2017-06-01 22:28:03 +08:00 |
|
yihua.huang
|
b879b0eed0
|
fix redisscheduler #583
|
2017-06-01 22:25:01 +08:00 |
|
Yihua Huang
|
9903f0367d
|
Merge pull request #570 from SoulZhong/master
修复formatter初始化未传参bug
|
2017-05-29 17:49:42 +08:00 |
|
yihua.huang
|
2e35e149be
|
for 0.7.1
|
2017-05-29 14:41:49 +08:00 |
|
yihua.huang
|
49de9374cd
|
new SimpleHttpClient #576
|
2017-05-27 17:30:19 +08:00 |
|
yihua.huang
|
7ffc6998ef
|
add isExtractLinks to OOSpider #575
|
2017-05-27 16:20:06 +08:00 |
soul
|
bc828e1384
|
修复formatter初始化未传参bug
|
2017-05-25 12:17:10 +08:00 |
|
yihua.huang
|
d38d51dfcb
|
fix javadoc
|
2017-04-15 12:24:50 +08:00 |
|
yihua.huang
|
1b04a7f2b3
|
#527 move logic check from downloaderto spider
|
2017-04-09 09:23:10 +08:00 |
|
yihua.huang
|
c13110c4cb
|
fix samples
|
2017-03-21 07:53:43 +08:00 |
|
yihua.huang
|
d87c73b472
|
change check-and-set to atomic sadd for redis DuplicateRemover #368
|
2017-03-01 22:24:34 +08:00 |
|
yihua.huang
|
3e633c6871
|
version
|
2017-01-21 11:51:14 +08:00 |
|
yihua.huang
|
f45e2f118b
|
for release
|
2017-01-21 11:38:36 +08:00 |
Jsbd
|
6d78d51fc0
|
Merge branch 'master' into master
|
2016-12-27 14:15:40 +08:00 |
|
yihua.huang
|
00dfebbceb
|
#424 remove guava dep and add fix docs
|
2016-12-18 10:45:50 +08:00 |
|
yihua.huang
|
a960a39c44
|
fix compile error for example change
|
2016-12-18 08:32:14 +08:00 |
|
yihua.huang
|
a3ee9e3d08
|
fix example
|
2016-12-18 08:18:26 +08:00 |
|
Jsbd
|
1b886d48a2
|
新增PhantomJSDownloader构造函数，支持crawl.js路径自定义，因为当其他项目依赖此jar包时，runtime.exec()执行phantomjs命令时无使用法jar包中的crawl.js
|
2016-12-08 14:29:42 +08:00 |
|
Jsbd
|
d1f2e65e5d
|
新增PhantomJSDownloader构造函数，支持crawl.js路径自定义，因为当其他项目依赖此jar包时，runtime.exec()执行phantomjs命令时无使用法jar包中的crawl.js
|
2016-12-08 14:28:48 +08:00 |
|
Jsbd
|
ebc61363c8
|
为PhantomJSDownloader添加新的构造函数，支持phantomjs自定义命令
为PhantomJSDownloader添加新的构造函数，支持phantomjs自定义命令
example:
* phantomjs.exe 支持windows环境
* phantomjs --ignore-ssl-errors=yes 忽略抓取地址是https时的一些错误
* /usr/local/bin/phantomjs 命令的绝对路径，避免因系统环境变量引起的IOException
|
2016-12-02 10:17:46 +08:00 |
|
yihua.huang
|
b92e6b04f0
|
#400 修复FileCacheQueueScheduler自己设置DuplicateRemover会导致NPE的问题
|
2016-11-25 08:30:24 +08:00 |
|
Yihua Huang
|
1491033534
|
Merge pull request #377 from jerry-sc/monitor-bug
fix the monitor bug which the spider will terminate when a seed url with port
|
2016-11-19 13:01:30 +08:00 |
Jerry
|
e56b8c3efc
|
fix the monitor bug which the spider will terminate when a seed url with port
|
2016-09-22 22:36:18 +08:00 |
郭玉昆
|
700898fe8a
|
fixed #301 修复使用注解抽取JSON数据的问题
|
2016-08-29 17:07:37 +08:00 |
Salon.sai
|
f89a6a6826
|
add: redis scheduler with priority
|
2016-07-05 16:29:01 +08:00 |
|
Yihua Huang
|
37cb43b667
|
Merge pull request #176 from lavenderx/master
add PhantomJSDownloader
|
2016-05-08 20:36:17 +08:00 |
|
yihua.huang
|
7edfa26f90
|
complete javadoc
|
2016-01-21 18:34:07 +08:00 |
|
yihua.huang
|
8b90b91e33
|
complete some javadoc
|
2016-01-21 18:14:10 +08:00 |
|
yihua.huang
|
9c5716a543
|
complete javadoc
|
2016-01-21 18:05:12 +08:00 |
|
yihua.huang
|
7586e3d75c
|
add some test for github repo downloader
|
2016-01-19 08:05:53 +08:00 |
|
Yihua Huang
|
cfde3b7657
|
Merge pull request #237 from SpenceZhou/master
Update RedisScheduler.java
|
2015-12-02 22:17:00 +08:00 |
SpenceZhou
|
165e5a72eb
|
Update RedisScheduler.java
修改redisscheduler中获取爬取总数bug
|
2015-12-02 17:10:42 +08:00 |
x1ny
|
90e14b31b0
|
修正FileCacheQueueScheduler导致程序不能正常结束和未关闭流
FileCacheQueueScheduler中开启了一个线程周期运行来保存数据但在爬虫结束后没有关闭导致程序无法结束，以及没有关闭io流。
解决方法：
让FileCacheQueueScheduler实现Closable接口，在close方法中关闭线程以及流。
在Spider的close方法中添加对scheduler的关闭操作。
|
2015-11-12 23:10:20 +08:00 |