yihua.huang
|
e5db538c19
|
#647 remove ThreadSafe annotation
|
2017-11-29 13:49:40 +08:00 |
|
yihua.huang
|
b539522ca8
|
#701 support to tls1.2
|
2017-11-29 13:36:19 +08:00 |
|
yihua.huang
|
e276b11ea9
|
vesion 0.7.3
|
2017-07-30 15:29:22 +08:00 |
王侃
|
a7e3090711
|
setBinaryContent, setCharset 方法返回this, 方便链式调用
|
2017-07-29 19:07:40 +08:00 |
|
yihua.huang
|
6f5b9e448e
|
#627 set charset to request
|
2017-07-29 11:27:56 +08:00 |
|
yihua.huang
|
32f1f2cf44
|
#613 add charset to page
|
2017-07-29 11:16:09 +08:00 |
|
yihua.huang
|
65049baca4
|
#571 fix spell mistake
|
2017-07-29 11:06:17 +08:00 |
|
yihua.huang
|
f375b9fdba
|
#629 fix ut
|
2017-07-25 18:27:10 +08:00 |
|
yihua.huang
|
3266ea15ca
|
#629 correct illegal url in HttpUriRequestConverter
|
2017-07-22 17:40:43 +08:00 |
|
yihua.huang
|
5daf92e8b2
|
#610 CASE_INSENSITIVE for charset detect in Content-Type
|
2017-07-22 17:27:36 +08:00 |
|
yihua.huang
|
2183ba9b61
|
#571 add getBytes to Page
|
2017-07-22 12:11:00 +08:00 |
|
yihua.huang
|
c3bdb20458
|
#631 remove IllegalArgumentException of HttpRequestBody.json and so on
|
2017-07-22 11:49:23 +08:00 |
|
yihua.huang
|
f405e642c0
|
add default constructor for HttpRequestBody #609
|
2017-06-24 11:31:17 +08:00 |
|
yihua.huang
|
eb376fca74
|
update jsoup to 1.10.3 #608
|
2017-06-24 11:29:35 +08:00 |
|
yihua.huang
|
faca38d4ec
|
update jsonpath to 2.2.0 #606
|
2017-06-24 11:01:23 +08:00 |
|
yihua.huang
|
1b6394bef9
|
version 0.7.2
|
2017-06-17 16:10:10 +08:00 |
|
yihua.huang
|
3ee00015c2
|
change header from Authorization to Proxy-Authorization for Proxy Authorization #596
|
2017-06-17 16:08:39 +08:00 |
|
yihua.huang
|
4111b07263
|
more error log on page code error #601
|
2017-06-17 10:36:41 +08:00 |
|
yihua.huang
|
6bd1eed25e
|
fix duplicate call of onSuccess and onError #605
|
2017-06-17 10:26:55 +08:00 |
|
yihua.huang
|
3a589d4ca6
|
HttpRequestBody implements Serializable #594
|
2017-06-08 15:59:46 +08:00 |
|
yihua.huang
|
13cdf82695
|
update version to 0.7.2-SNAPSHOT
|
2017-06-05 16:55:59 +08:00 |
|
yihua.huang
|
1e9187f24e
|
version 0.7.1
|
2017-06-04 07:52:54 +08:00 |
|
yihua.huang
|
592fa2c0f1
|
add site header test
|
2017-06-03 11:56:18 +08:00 |
|
yihua.huang
|
19d34dbb65
|
not add bracket to regex in RegexSelector #559
|
2017-06-03 11:30:07 +08:00 |
|
Yihua Huang
|
b6b991a09b
|
Merge pull request #556 from zhuyuesut/master
增加对零宽断言的支持
|
2017-06-03 11:10:56 +08:00 |
|
yihua.huang
|
bb0eb69acf
|
update ZhihuPageProcessor example
|
2017-06-03 10:42:17 +08:00 |
|
yihua.huang
|
2e35e149be
|
for 0.7.1
|
2017-05-29 14:41:49 +08:00 |
|
yihua.huang
|
17d8bfa907
|
docs and pgp version
|
2017-05-29 09:36:42 +08:00 |
|
yihua.huang
|
17478fcfc4
|
0.7.0 release
|
2017-05-29 09:30:56 +08:00 |
|
yihua.huang
|
636359300f
|
add Site.disableCookieManagement #577
|
2017-05-29 08:29:53 +08:00 |
|
yihua.huang
|
49de9374cd
|
new SimpleHttpClient #576
|
2017-05-27 17:30:19 +08:00 |
|
yihua.huang
|
8999ea9320
|
add public constructor for SimpleProxyProvider
|
2017-05-27 16:09:02 +08:00 |
|
yihua.huang
|
a8c2e6c729
|
alpha release
|
2017-05-20 12:51:16 +08:00 |
|
yihua.huang
|
3c1338193b
|
for 0.7.0.alpha
|
2017-05-20 12:34:09 +08:00 |
|
yihua.huang
|
e8abc28072
|
#552 add some log when crawler stop
|
2017-05-20 11:44:51 +08:00 |
zhuyue
|
9e1b7ed3f7
|
Update RegexSelector.java
|
2017-05-05 10:47:10 +08:00 |
|
zhuyue
|
c80f25edbd
|
Update RegexSelectorTest.java
简单的增加了一点测试
|
2017-05-03 18:33:23 +08:00 |
|
zhuyue
|
c3183252ac
|
Update RegexSelector.java
|
2017-05-03 18:24:19 +08:00 |
|
yihua.huang
|
cbf80af5dd
|
test for SimpleProxyProvider #535
|
2017-04-16 10:50:27 +08:00 |
|
yihua.huang
|
eb632a93d3
|
SimpleProxyProvider #535
|
2017-04-16 10:43:56 +08:00 |
|
yihua.huang
|
d38d51dfcb
|
fix javadoc
|
2017-04-15 12:24:50 +08:00 |
GZhY
|
5f34adf938
|
完善 LinksSelector.selectList 的测试用例
|
2017-04-09 21:29:01 +08:00 |
|
GZhY
|
ce3f0ac239
|
删除 fixAllRelativeHrefs 并修复 SeleniumDownloader 对 fixAllRelativeHrefs 的依赖
|
2017-04-09 21:01:32 +08:00 |
|
GZhY
|
bc6e81e00f
|
修复checkElementAndConvert方法注释中注释错误
|
2017-04-09 20:40:00 +08:00 |
|
yihua.huang
|
4a2c0f4f97
|
add returnProxy for proxyProvider
|
2017-04-09 09:28:36 +08:00 |
|
yihua.huang
|
1b04a7f2b3
|
#527 move logic check from downloaderto spider
|
2017-04-09 09:23:10 +08:00 |
|
yihua.huang
|
0f4d6e8b12
|
#525 remove port in UrlUtils.getDomain()
|
2017-04-08 23:17:00 +08:00 |
|
yihua.huang
|
a1ae632b62
|
test for request cookies and headers
|
2017-04-08 23:13:16 +08:00 |
|
yihua.huang
|
db67db8103
|
#523 remove fixAllRelativeHrefs by default, get absolute urls for links()
|
2017-04-08 22:06:18 +08:00 |
|
yihua.huang
|
abd020b45b
|
some comments
|
2017-04-08 20:16:17 +08:00 |
|
yihua.huang
|
2622b448b8
|
fix test
|
2017-04-08 20:09:43 +08:00 |
|
yihua.huang
|
b06a248c00
|
fix test
|
2017-04-08 20:06:04 +08:00 |
|
yihua.huang
|
1cfbd13aae
|
refacor in httpclientdownloader
|
2017-04-08 20:04:56 +08:00 |
|
yihua.huang
|
83ada9749e
|
fix test
|
2017-04-08 12:16:34 +08:00 |
|
yihua.huang
|
fe95a6842f
|
Request再次重构:去掉params，仅保留HttpRequestBody
|
2017-04-08 12:12:39 +08:00 |
|
yihua.huang
|
395396c68e
|
增加HttpRequestBody
|
2017-04-08 11:59:52 +08:00 |
xbynet
|
c93a8a2722
|
修复字符编码检测BUG
|
2017-03-31 18:27:18 +08:00 |
|
yihua.huang
|
74110e6ec5
|
remove useless file
|
2017-03-30 09:19:00 +08:00 |
|
yihua.huang
|
b100dfe273
|
update version
|
2017-03-30 09:18:12 +08:00 |
|
xbynet@outlook.com
|
1c24baa8d1
|
Request支持设置header与cookie
新增POST请求时,XML、JSON参数支持
Page支持获取响应header
|
2017-03-29 02:27:06 +08:00 |
|
yihua.huang
|
6bd197859b
|
fix test
|
2017-03-27 09:55:50 +08:00 |
|
yihua.huang
|
f23e138c72
|
add response headers to Page #508
|
2017-03-27 09:52:25 +08:00 |
|
yihua.huang
|
c13110c4cb
|
fix samples
|
2017-03-21 07:53:43 +08:00 |
|
yihua.huang
|
c51ac6017c
|
remove Site.addStartRequest() etc. #494
|
2017-03-21 07:32:34 +08:00 |
|
yihua.huang
|
68050fc88e
|
test pass
|
2017-03-21 07:19:12 +08:00 |
|
yihua.huang
|
474b7c9d57
|
refactor
|
2017-03-20 23:13:49 +08:00 |
|
yihua.huang
|
25c81013ca
|
new proxy pool api
|
2017-03-20 22:47:24 +08:00 |
|
yihua.huang
|
46297deaa1
|
HttpUriRequestConverter
|
2017-03-20 22:43:32 +08:00 |
|
yihua.huang
|
1d86f7c048
|
compile passed in httpclientDownloader
|
2017-03-20 22:40:14 +08:00 |
|
yihua.huang
|
b71f379512
|
fix
|
2017-03-18 12:18:00 +08:00 |
|
yihua.huang
|
a7f9e7cad5
|
重构一部分httpclient
|
2017-03-18 12:16:21 +08:00 |
|
yihua.huang
|
221c155060
|
move release connection before return proxy #396
|
2017-03-18 11:15:36 +08:00 |
|
yihua.huang
|
68beff42c5
|
add test #493
|
2017-03-18 11:01:30 +08:00 |
wuyifan
|
79522f941e
|
Bug, add null check to site in HttpClientDownloader & HttpClientGenerator
|
2017-03-17 14:10:54 +08:00 |
|
yihua.huang
|
e9341d0291
|
complete test #447
|
2017-03-17 07:54:28 +08:00 |
|
yihua.huang
|
e7d35c4846
|
add params to all method of request #447
|
2017-03-17 07:18:05 +08:00 |
|
yihua.huang
|
75bad591d7
|
rewrite hashCode and equals for params #447
|
2017-03-17 07:10:14 +08:00 |
|
Yihua Huang
|
11c32669b2
|
Merge pull request #447 from xbynet/master
简化POST参数设置.
|
2017-03-17 07:06:46 +08:00 |
|
yihua.huang
|
aa01e27779
|
change constructor for Proxy to public #490
|
2017-03-17 07:02:02 +08:00 |
mei
|
791520e6a0
|
fix a bug of RegexSelector when regex has zero-width assertions.
|
2017-03-17 00:06:15 +08:00 |
|
yihua.huang
|
c175ea88c0
|
#more test #484
|
2017-03-11 11:43:18 +08:00 |
|
yihua.huang
|
9b964c0a99
|
test for #484
|
2017-03-11 11:41:01 +08:00 |
|
yihua.huang
|
fc702fd3b6
|
introduce mockito for test
|
2017-03-11 11:31:15 +08:00 |
|
yihua.huang
|
5215a492cc
|
remove duplicate check for POST request #484
|
2017-03-11 11:26:13 +08:00 |
|
yihua.huang
|
0a1fb19052
|
add tests #483
|
2017-03-11 10:56:31 +08:00 |
|
yihua.huang
|
a2e7f0004b
|
Merge branch 'master' of github.com:code4craft/webmagic
|
2017-03-11 10:52:54 +08:00 |
|
yihua.huang
|
ef32571821
|
rewrite Request.equals and hashCode, add Method to check #483
|
2017-03-11 10:52:39 +08:00 |
|
yihua.huang
|
8b8f535c30
|
refactor:extract charset detect to utils
|
2017-03-11 10:43:10 +08:00 |
Ckex.zha
|
e645524ad2
|
fix bug,set ExecutorService
|
2017-03-04 20:57:29 +08:00 |
|
yihua.huang
|
a872a6480e
|
fix code sample for github #348
|
2017-02-25 22:46:29 +08:00 |
|
yihua.huang
|
1d2171805f
|
add test for #228
|
2017-02-25 22:30:48 +08:00 |
|
yihua.huang
|
bbe0b52ddd
|
remove synchronized in QueueScheduler #410
|
2017-02-25 19:55:45 +08:00 |
|
yihua.huang
|
ad69963005
|
remove synchronize in Page #411
|
2017-02-25 19:42:12 +08:00 |
|
yihua.huang
|
3a796b9413
|
remove duplicate code #421
|
2017-02-25 12:01:12 +08:00 |
|
yihua.huang
|
42f1018010
|
remove messy code
|
2017-02-21 14:08:05 +08:00 |
|
xbynet
|
650468c0e4
|
解决POST中文参数乱码问题
|
2017-01-22 18:04:22 +08:00 |
|
yihua.huang
|
aaccc93215
|
new version
|
2017-01-21 12:04:12 +08:00 |
|
yihua.huang
|
3e633c6871
|
version
|
2017-01-21 11:51:14 +08:00 |
|
yihua.huang
|
f45e2f118b
|
for release
|
2017-01-21 11:38:36 +08:00 |
|
yihua.huang
|
d60615f503
|
修复使用startUrls没有设置domain导致使用cookie空指针的问题#438
|
2017-01-21 11:29:42 +08:00 |
|
yihua.huang
|
407fbb6130
|
refactor logger#445
|
2017-01-21 11:05:54 +08:00 |
|
Ckex.zha
|
0dc26c8ca0
|
optimize code.
|
2017-01-20 14:03:26 +08:00 |
|
Yihua Huang
|
4f76d62d4f
|
Merge pull request #444 from ckex/develop
绕过安全证书
|
2017-01-18 23:42:51 +08:00 |
|
Ckex.zha
|
e4af05a6f2
|
绕过安全证书
|
2017-01-18 17:28:01 +08:00 |
|
xbynet@outlook.com
|
c23627bf63
|
解决post/redirect/post 302跳转问题
|
2017-01-17 00:07:01 +08:00 |
|
yihua.huang
|
d69204b919
|
0.6.0
|
2016-12-18 11:45:43 +08:00 |
|
yihua.huang
|
9bdb48b2d0
|
version 0.6.0
|
2016-12-18 11:20:28 +08:00 |
|
yihua.huang
|
eeb607fd0e
|
将Spider.processRequest()抛出异常改回原来的逻辑
|
2016-12-18 11:04:58 +08:00 |
|
yihua.huang
|
97592d6720
|
Version 0.6.0
|
2016-12-18 10:58:24 +08:00 |
|
yihua.huang
|
00dfebbceb
|
#424 remove guava dep and add fix docs
|
2016-12-18 10:45:50 +08:00 |
|
yihua.huang
|
c2531c6817
|
clean dependency
|
2016-12-18 08:34:46 +08:00 |
|
yihua.huang
|
a960a39c44
|
fix compile error for example change
|
2016-12-18 08:32:14 +08:00 |
|
yihua.huang
|
7476ceccee
|
more stable test
|
2016-12-18 08:15:26 +08:00 |
|
yihua.huang
|
5ce3fdfe5a
|
some refactor in log
|
2016-12-18 08:15:09 +08:00 |
|
yihua.huang
|
98163a3e40
|
update examples
|
2016-12-18 07:46:18 +08:00 |
|
yihua.huang
|
b090dcd20d
|
sepcific error page for HttpClientDownloaderTest to avoid test error when local port is available
|
2016-12-18 07:15:06 +08:00 |
|
yihua.huang
|
8f942d6fe2
|
#419 修复抓取https链接线程无法结束导致进程一直运行的问题
|
2016-12-18 06:56:01 +08:00 |
|
yihua.huang
|
dafd2b77ff
|
fix GithubRepoPageProcessor in example
|
2016-11-24 08:18:06 +08:00 |
|
yihua.huang
|
cfed860fb9
|
Merge branch 'master' of github.com:code4craft/webmagic
|
2016-11-22 17:00:27 +08:00 |
|
yihua.huang
|
2189aab652
|
fix test
|
2016-11-22 16:58:49 +08:00 |
|
Yihua Huang
|
1491033534
|
Merge pull request #377 from jerry-sc/monitor-bug
fix the monitor bug which the spider will terminate when a seed url with port
|
2016-11-19 13:01:30 +08:00 |
|
yihua.huang
|
507556d0aa
|
fix test: ProxyTest.testProxy() do not load exist proxy config
|
2016-11-19 12:54:39 +08:00 |
Jerry
|
e56b8c3efc
|
fix the monitor bug which the spider will terminate when a seed url with port
|
2016-09-22 22:36:18 +08:00 |
|
yihua.huang
|
448e528140
|
update StringUtils to apache lang3 #314
|
2016-05-24 13:33:17 +08:00 |
|
yihua.huang
|
3e33959b7a
|
#319 fix javadoc
|
2016-05-24 13:17:35 +08:00 |
|
yihua.huang
|
8730e3e97a
|
Merge branch 'fix' of git://github.com/kapsterio/webmagic into kapsterio-fix
|
2016-05-08 20:46:22 +08:00 |
|
yihua.huang
|
2400ff7e1a
|
resovle conflict
|
2016-05-08 20:31:43 +08:00 |
|
yihua.huang
|
b7f3c4bba0
|
Merge branch 'master' of git://github.com/hepan/webmagic into hepan-master
|
2016-05-08 20:27:47 +08:00 |
|
yihua.huang
|
d8f978fd20
|
fix test in JsonPathSelectorTest #289
|
2016-05-08 19:32:03 +08:00 |
|
yihua.huang
|
61c28a0130
|
refactor on proxypool
|
2016-05-08 17:53:15 +08:00 |
|
yihua.huang
|
b871b210c5
|
Merge branch 'proxy-strategy' of github.com:EdwardsBean/webmagic into EdwardsBean-proxy-strategy
|
2016-05-08 17:53:02 +08:00 |
|
yihua.huang
|
b5413368de
|
update ut
|
2016-05-08 16:23:41 +08:00 |
Jon
|
83c27ebbc4
|
增加IP代理认证功能
|
2016-05-08 16:17:58 +08:00 |
|
yihua.huang
|
ca072c5575
|
fix URL regex in GithubRepoPageProcessor #305
|
2016-05-08 12:09:45 +08:00 |
hepan
|
89c6e52863
|
代理增加用户名密码认证
|
2016-04-13 15:16:57 +08:00 |
Linker Lin
|
047cb8ff8f
|
updated versions to 0.5.4-SNAPSHOT
|
2016-04-01 14:51:59 +08:00 |
zhangheng09
|
6b179c3d55
|
这个改动的原因基于两点：1）代理归还给代理池的时机应该是执行完http请求后就要尽早归还 2)http代理应该是HttpClientDownloader该考虑的事，不应该有Spider来处理，Spider并不知道它的downloader是个HttpClientDownloader
|
2016-03-12 20:09:41 +08:00 |
|
zhangheng09
|
5f106c9c69
|
当page为null时，意味着非正常的响应状态，应该抛出异常，否则SpiderListener的onSuccess方法和onError方法都会执行
|
2016-03-12 20:03:27 +08:00 |
|
yihua.huang
|
c0b8e8f8ae
|
remove .classpath .project
|
2016-01-22 14:58:22 +08:00 |
|
yihua.huang
|
a8e6de4b90
|
Merge branch 'master' of git.oschina.net:flashsword20/webmagic
|
2016-01-22 10:16:58 +08:00 |
|
yihua.huang
|
0fd4623f0a
|
Merge branch 'osc'
|
2016-01-21 19:33:30 +08:00 |
|
yihua.huang
|
ce5495ecd5
|
remove useless files
|
2016-01-21 19:31:50 +08:00 |
|
yihua.huang
|
8265c7dade
|
remove submodules for relase
|
2016-01-21 19:25:13 +08:00 |
|
yihua.huang
|
7edfa26f90
|
complete javadoc
|
2016-01-21 18:34:07 +08:00 |
|
yihua.huang
|
8b90b91e33
|
complete some javadoc
|
2016-01-21 18:14:10 +08:00 |
|
yihua.huang
|
2b556cf053
|
update verison to 0.5.3-SNAPSHOT
|
2016-01-21 18:05:56 +08:00 |
|
yihua.huang
|
9c5716a543
|
complete javadoc
|
2016-01-21 18:05:12 +08:00 |
|
yihua.huang
|
db3cbf6ca5
|
update version to 0.5.3-SNAPSHOT
|
2016-01-21 17:58:36 +08:00 |
|
yihua.huang
|
81ce1ffc5f
|
fix ignore
|
2016-01-21 12:36:49 +08:00 |
|
yihua.huang
|
93764fa2c9
|
ignore some test
|
2016-01-21 12:28:32 +08:00 |