xbynet
|
c93a8a2722
|
修复字符编码检测BUG
|
2017-03-31 18:27:18 +08:00 |
|
xbynet@outlook.com
|
1c24baa8d1
|
Request支持设置header与cookie
新增POST请求时,XML、JSON参数支持
Page支持获取响应header
|
2017-03-29 02:27:06 +08:00 |
yihua.huang
|
221c155060
|
move release connection before return proxy #396
|
2017-03-18 11:15:36 +08:00 |
wuyifan
|
79522f941e
|
Bug, add null check to site in HttpClientDownloader & HttpClientGenerator
|
2017-03-17 14:10:54 +08:00 |
|
yihua.huang
|
e9341d0291
|
complete test #447
|
2017-03-17 07:54:28 +08:00 |
|
yihua.huang
|
e7d35c4846
|
add params to all method of request #447
|
2017-03-17 07:18:05 +08:00 |
|
yihua.huang
|
75bad591d7
|
rewrite hashCode and equals for params #447
|
2017-03-17 07:10:14 +08:00 |
|
Yihua Huang
|
11c32669b2
|
Merge pull request #447 from xbynet/master
简化POST参数设置.
|
2017-03-17 07:06:46 +08:00 |
|
yihua.huang
|
aa01e27779
|
change constructor for Proxy to public #490
|
2017-03-17 07:02:02 +08:00 |
mei
|
791520e6a0
|
fix a bug of RegexSelector when regex has zero-width assertions.
|
2017-03-17 00:06:15 +08:00 |
|
yihua.huang
|
9b964c0a99
|
test for #484
|
2017-03-11 11:41:01 +08:00 |
|
yihua.huang
|
5215a492cc
|
remove duplicate check for POST request #484
|
2017-03-11 11:26:13 +08:00 |
|
yihua.huang
|
a2e7f0004b
|
Merge branch 'master' of github.com:code4craft/webmagic
|
2017-03-11 10:52:54 +08:00 |
|
yihua.huang
|
ef32571821
|
rewrite Request.equals and hashCode, add Method to check #483
|
2017-03-11 10:52:39 +08:00 |
|
yihua.huang
|
8b8f535c30
|
refactor:extract charset detect to utils
|
2017-03-11 10:43:10 +08:00 |
Ckex.zha
|
e645524ad2
|
fix bug,set ExecutorService
|
2017-03-04 20:57:29 +08:00 |
|
yihua.huang
|
a872a6480e
|
fix code sample for github #348
|
2017-02-25 22:46:29 +08:00 |
|
yihua.huang
|
bbe0b52ddd
|
remove synchronized in QueueScheduler #410
|
2017-02-25 19:55:45 +08:00 |
|
yihua.huang
|
ad69963005
|
remove synchronize in Page #411
|
2017-02-25 19:42:12 +08:00 |
|
yihua.huang
|
3a796b9413
|
remove duplicate code #421
|
2017-02-25 12:01:12 +08:00 |
|
yihua.huang
|
42f1018010
|
remove messy code
|
2017-02-21 14:08:05 +08:00 |
|
xbynet
|
650468c0e4
|
解决POST中文参数乱码问题
|
2017-01-22 18:04:22 +08:00 |
|
yihua.huang
|
d60615f503
|
修复使用startUrls没有设置domain导致使用cookie空指针的问题#438
|
2017-01-21 11:29:42 +08:00 |
|
yihua.huang
|
407fbb6130
|
refactor logger#445
|
2017-01-21 11:05:54 +08:00 |
|
Ckex.zha
|
0dc26c8ca0
|
optimize code.
|
2017-01-20 14:03:26 +08:00 |
|
Yihua Huang
|
4f76d62d4f
|
Merge pull request #444 from ckex/develop
绕过安全证书
|
2017-01-18 23:42:51 +08:00 |
|
Ckex.zha
|
e4af05a6f2
|
绕过安全证书
|
2017-01-18 17:28:01 +08:00 |
|
xbynet@outlook.com
|
c23627bf63
|
解决post/redirect/post 302跳转问题
|
2017-01-17 00:07:01 +08:00 |
|
yihua.huang
|
9bdb48b2d0
|
version 0.6.0
|
2016-12-18 11:20:28 +08:00 |
|
yihua.huang
|
eeb607fd0e
|
将Spider.processRequest()抛出异常改回原来的逻辑
|
2016-12-18 11:04:58 +08:00 |
|
yihua.huang
|
00dfebbceb
|
#424 remove guava dep and add fix docs
|
2016-12-18 10:45:50 +08:00 |
|
yihua.huang
|
5ce3fdfe5a
|
some refactor in log
|
2016-12-18 08:15:09 +08:00 |
|
yihua.huang
|
98163a3e40
|
update examples
|
2016-12-18 07:46:18 +08:00 |
|
yihua.huang
|
8f942d6fe2
|
#419 修复抓取https链接线程无法结束导致进程一直运行的问题
|
2016-12-18 06:56:01 +08:00 |
|
yihua.huang
|
dafd2b77ff
|
fix GithubRepoPageProcessor in example
|
2016-11-24 08:18:06 +08:00 |
Jerry
|
e56b8c3efc
|
fix the monitor bug which the spider will terminate when a seed url with port
|
2016-09-22 22:36:18 +08:00 |
|
yihua.huang
|
448e528140
|
update StringUtils to apache lang3 #314
|
2016-05-24 13:33:17 +08:00 |
|
yihua.huang
|
3e33959b7a
|
#319 fix javadoc
|
2016-05-24 13:17:35 +08:00 |
|
yihua.huang
|
8730e3e97a
|
Merge branch 'fix' of git://github.com/kapsterio/webmagic into kapsterio-fix
|
2016-05-08 20:46:22 +08:00 |
|
yihua.huang
|
2400ff7e1a
|
resovle conflict
|
2016-05-08 20:31:43 +08:00 |
|
yihua.huang
|
b7f3c4bba0
|
Merge branch 'master' of git://github.com/hepan/webmagic into hepan-master
|
2016-05-08 20:27:47 +08:00 |
|
yihua.huang
|
61c28a0130
|
refactor on proxypool
|
2016-05-08 17:53:15 +08:00 |
|
yihua.huang
|
b871b210c5
|
Merge branch 'proxy-strategy' of github.com:EdwardsBean/webmagic into EdwardsBean-proxy-strategy
|
2016-05-08 17:53:02 +08:00 |
Jon
|
83c27ebbc4
|
增加IP代理认证功能
|
2016-05-08 16:17:58 +08:00 |
|
yihua.huang
|
ca072c5575
|
fix URL regex in GithubRepoPageProcessor #305
|
2016-05-08 12:09:45 +08:00 |
hepan
|
89c6e52863
|
代理增加用户名密码认证
|
2016-04-13 15:16:57 +08:00 |
zhangheng09
|
6b179c3d55
|
这个改动的原因基于两点：1）代理归还给代理池的时机应该是执行完http请求后就要尽早归还 2)http代理应该是HttpClientDownloader该考虑的事，不应该有Spider来处理，Spider并不知道它的downloader是个HttpClientDownloader
|
2016-03-12 20:09:41 +08:00 |
|
zhangheng09
|
5f106c9c69
|
当page为null时，意味着非正常的响应状态，应该抛出异常，否则SpiderListener的onSuccess方法和onError方法都会执行
|
2016-03-12 20:03:27 +08:00 |
|
yihua.huang
|
7edfa26f90
|
complete javadoc
|
2016-01-21 18:34:07 +08:00 |
|
yihua.huang
|
8b90b91e33
|
complete some javadoc
|
2016-01-21 18:14:10 +08:00 |