3ee00015c2
change header from Authorization to Proxy-Authorization for Proxy Authorization #596
2017-06-17 16:08:39 +08:00
4111b07263
more error log on page code error #601
2017-06-17 10:36:41 +08:00
6bd1eed25e
fix duplicate call of onSuccess and onError #605
2017-06-17 10:26:55 +08:00
3a589d4ca6
HttpRequestBody implements Serializable #594
2017-06-08 15:59:46 +08:00
13cdf82695
update version to 0.7.2-SNAPSHOT
2017-06-05 16:55:59 +08:00
1e9187f24e
version 0.7.1
2017-06-04 07:52:54 +08:00
592fa2c0f1
add site header test
2017-06-03 11:56:18 +08:00
19d34dbb65
not add bracket to regex in RegexSelector #559
2017-06-03 11:30:07 +08:00
b6b991a09b
Merge pull request #556 from zhuyuesut/master
...
增加对零宽断言的支持
2017-06-03 11:10:56 +08:00
bb0eb69acf
update ZhihuPageProcessor example
2017-06-03 10:42:17 +08:00
2e35e149be
for 0.7.1
2017-05-29 14:41:49 +08:00
17d8bfa907
docs and pgp version
2017-05-29 09:36:42 +08:00
17478fcfc4
0.7.0 release
2017-05-29 09:30:56 +08:00
636359300f
add Site.disableCookieManagement #577
2017-05-29 08:29:53 +08:00
49de9374cd
new SimpleHttpClient #576
2017-05-27 17:30:19 +08:00
8999ea9320
add public constructor for SimpleProxyProvider
2017-05-27 16:09:02 +08:00
a8c2e6c729
alpha release
2017-05-20 12:51:16 +08:00
3c1338193b
for 0.7.0.alpha
2017-05-20 12:34:09 +08:00
e8abc28072
#552 add some log when crawler stop
2017-05-20 11:44:51 +08:00
9e1b7ed3f7
Update RegexSelector.java
2017-05-05 10:47:10 +08:00
c80f25edbd
Update RegexSelectorTest.java
...
简单的增加了一点测试
2017-05-03 18:33:23 +08:00
c3183252ac
Update RegexSelector.java
2017-05-03 18:24:19 +08:00
cbf80af5dd
test for SimpleProxyProvider #535
2017-04-16 10:50:27 +08:00
eb632a93d3
SimpleProxyProvider #535
2017-04-16 10:43:56 +08:00
d38d51dfcb
fix javadoc
2017-04-15 12:24:50 +08:00
5f34adf938
完善 LinksSelector.selectList 的测试用例
2017-04-09 21:29:01 +08:00
ce3f0ac239
删除 fixAllRelativeHrefs 并修复 SeleniumDownloader 对 fixAllRelativeHrefs 的依赖
2017-04-09 21:01:32 +08:00
bc6e81e00f
修复checkElementAndConvert方法注释中注释错误
2017-04-09 20:40:00 +08:00
4a2c0f4f97
add returnProxy for proxyProvider
2017-04-09 09:28:36 +08:00
1b04a7f2b3
#527 move logic check from downloaderto spider
2017-04-09 09:23:10 +08:00
0f4d6e8b12
#525 remove port in UrlUtils.getDomain()
2017-04-08 23:17:00 +08:00
a1ae632b62
test for request cookies and headers
2017-04-08 23:13:16 +08:00
db67db8103
#523 remove fixAllRelativeHrefs by default, get absolute urls for links()
2017-04-08 22:06:18 +08:00
abd020b45b
some comments
2017-04-08 20:16:17 +08:00
2622b448b8
fix test
2017-04-08 20:09:43 +08:00
b06a248c00
fix test
2017-04-08 20:06:04 +08:00
1cfbd13aae
refacor in httpclientdownloader
2017-04-08 20:04:56 +08:00
83ada9749e
fix test
2017-04-08 12:16:34 +08:00
fe95a6842f
Request再次重构:去掉params，仅保留HttpRequestBody
2017-04-08 12:12:39 +08:00
395396c68e
增加HttpRequestBody
2017-04-08 11:59:52 +08:00
c93a8a2722
修复字符编码检测BUG
2017-03-31 18:27:18 +08:00
74110e6ec5
remove useless file
2017-03-30 09:19:00 +08:00
b100dfe273
update version
2017-03-30 09:18:12 +08:00
1c24baa8d1
Request支持设置header与cookie
...
新增POST请求时,XML、JSON参数支持
Page支持获取响应header
2017-03-29 02:27:06 +08:00
6bd197859b
fix test
2017-03-27 09:55:50 +08:00
f23e138c72
add response headers to Page #508
2017-03-27 09:52:25 +08:00
c13110c4cb
fix samples
2017-03-21 07:53:43 +08:00
c51ac6017c
remove Site.addStartRequest() etc. #494
2017-03-21 07:32:34 +08:00
68050fc88e
test pass
2017-03-21 07:19:12 +08:00
474b7c9d57
refactor
2017-03-20 23:13:49 +08:00
25c81013ca
new proxy pool api
2017-03-20 22:47:24 +08:00
46297deaa1
HttpUriRequestConverter
2017-03-20 22:43:32 +08:00
1d86f7c048
compile passed in httpclientDownloader
2017-03-20 22:40:14 +08:00
b71f379512
fix
2017-03-18 12:18:00 +08:00
a7f9e7cad5
重构一部分httpclient
2017-03-18 12:16:21 +08:00
221c155060
move release connection before return proxy #396
2017-03-18 11:15:36 +08:00
68beff42c5
add test #493
2017-03-18 11:01:30 +08:00
79522f941e
Bug, add null check to site in HttpClientDownloader & HttpClientGenerator
2017-03-17 14:10:54 +08:00
e9341d0291
complete test #447
2017-03-17 07:54:28 +08:00
e7d35c4846
add params to all method of request #447
2017-03-17 07:18:05 +08:00
75bad591d7
rewrite hashCode and equals for params #447
2017-03-17 07:10:14 +08:00
11c32669b2
Merge pull request #447 from xbynet/master
...
简化POST参数设置.
2017-03-17 07:06:46 +08:00
aa01e27779
change constructor for Proxy to public #490
2017-03-17 07:02:02 +08:00
791520e6a0
fix a bug of RegexSelector when regex has zero-width assertions.
2017-03-17 00:06:15 +08:00
c175ea88c0
#more test #484
2017-03-11 11:43:18 +08:00
9b964c0a99
test for #484
2017-03-11 11:41:01 +08:00
fc702fd3b6
introduce mockito for test
2017-03-11 11:31:15 +08:00
5215a492cc
remove duplicate check for POST request #484
2017-03-11 11:26:13 +08:00
0a1fb19052
add tests #483
2017-03-11 10:56:31 +08:00
a2e7f0004b
Merge branch 'master' of github.com:code4craft/webmagic
2017-03-11 10:52:54 +08:00
ef32571821
rewrite Request.equals and hashCode, add Method to check #483
2017-03-11 10:52:39 +08:00
8b8f535c30
refactor:extract charset detect to utils
2017-03-11 10:43:10 +08:00
e645524ad2
fix bug,set ExecutorService
2017-03-04 20:57:29 +08:00
a872a6480e
fix code sample for github #348
2017-02-25 22:46:29 +08:00
1d2171805f
add test for #228
2017-02-25 22:30:48 +08:00
bbe0b52ddd
remove synchronized in QueueScheduler #410
2017-02-25 19:55:45 +08:00
ad69963005
remove synchronize in Page #411
2017-02-25 19:42:12 +08:00
3a796b9413
remove duplicate code #421
2017-02-25 12:01:12 +08:00
42f1018010
remove messy code
2017-02-21 14:08:05 +08:00
650468c0e4
解决POST中文参数乱码问题
2017-01-22 18:04:22 +08:00
aaccc93215
new version
2017-01-21 12:04:12 +08:00
3e633c6871
version
2017-01-21 11:51:14 +08:00
f45e2f118b
for release
2017-01-21 11:38:36 +08:00
d60615f503
修复使用startUrls没有设置domain导致使用cookie空指针的问题#438
2017-01-21 11:29:42 +08:00
407fbb6130
refactor logger#445
2017-01-21 11:05:54 +08:00
0dc26c8ca0
optimize code.
2017-01-20 14:03:26 +08:00
4f76d62d4f
Merge pull request #444 from ckex/develop
...
绕过安全证书
2017-01-18 23:42:51 +08:00
e4af05a6f2
绕过安全证书
2017-01-18 17:28:01 +08:00
c23627bf63
解决post/redirect/post 302跳转问题
2017-01-17 00:07:01 +08:00
d69204b919
0.6.0
2016-12-18 11:45:43 +08:00
9bdb48b2d0
version 0.6.0
2016-12-18 11:20:28 +08:00
eeb607fd0e
将Spider.processRequest()抛出异常改回原来的逻辑
2016-12-18 11:04:58 +08:00
97592d6720
Version 0.6.0
2016-12-18 10:58:24 +08:00
00dfebbceb
#424 remove guava dep and add fix docs
2016-12-18 10:45:50 +08:00
c2531c6817
clean dependency
2016-12-18 08:34:46 +08:00
a960a39c44
fix compile error for example change
2016-12-18 08:32:14 +08:00
7476ceccee
more stable test
2016-12-18 08:15:26 +08:00
5ce3fdfe5a
some refactor in log
2016-12-18 08:15:09 +08:00
98163a3e40
update examples
2016-12-18 07:46:18 +08:00
b090dcd20d
sepcific error page for HttpClientDownloaderTest to avoid test error when local port is available
2016-12-18 07:15:06 +08:00
8f942d6fe2
#419 修复抓取https链接线程无法结束导致进程一直运行的问题
2016-12-18 06:56:01 +08:00
dafd2b77ff
fix GithubRepoPageProcessor in example
2016-11-24 08:18:06 +08:00
cfed860fb9
Merge branch 'master' of github.com:code4craft/webmagic
2016-11-22 17:00:27 +08:00
2189aab652
fix test
2016-11-22 16:58:49 +08:00
1491033534
Merge pull request #377 from jerry-sc/monitor-bug
...
fix the monitor bug which the spider will terminate when a seed url with port
2016-11-19 13:01:30 +08:00
507556d0aa
fix test: ProxyTest.testProxy() do not load exist proxy config
2016-11-19 12:54:39 +08:00
e56b8c3efc
fix the monitor bug which the spider will terminate when a seed url with port
2016-09-22 22:36:18 +08:00
448e528140
update StringUtils to apache lang3 #314
2016-05-24 13:33:17 +08:00
3e33959b7a
#319 fix javadoc
2016-05-24 13:17:35 +08:00
8730e3e97a
Merge branch 'fix' of git://github.com/kapsterio/webmagic into kapsterio-fix
2016-05-08 20:46:22 +08:00
2400ff7e1a
resovle conflict
2016-05-08 20:31:43 +08:00
b7f3c4bba0
Merge branch 'master' of git://github.com/hepan/webmagic into hepan-master
2016-05-08 20:27:47 +08:00
d8f978fd20
fix test in JsonPathSelectorTest #289
2016-05-08 19:32:03 +08:00
61c28a0130
refactor on proxypool
2016-05-08 17:53:15 +08:00
b871b210c5
Merge branch 'proxy-strategy' of github.com:EdwardsBean/webmagic into EdwardsBean-proxy-strategy
2016-05-08 17:53:02 +08:00
b5413368de
update ut
2016-05-08 16:23:41 +08:00
83c27ebbc4
增加IP代理认证功能
2016-05-08 16:17:58 +08:00
ca072c5575
fix URL regex in GithubRepoPageProcessor #305
2016-05-08 12:09:45 +08:00
89c6e52863
代理增加用户名密码认证
2016-04-13 15:16:57 +08:00
047cb8ff8f
updated versions to 0.5.4-SNAPSHOT
2016-04-01 14:51:59 +08:00
6b179c3d55
这个改动的原因基于两点：1）代理归还给代理池的时机应该是执行完http请求后就要尽早归还 2)http代理应该是HttpClientDownloader该考虑的事，不应该有Spider来处理，Spider并不知道它的downloader是个HttpClientDownloader
2016-03-12 20:09:41 +08:00
5f106c9c69
当page为null时，意味着非正常的响应状态，应该抛出异常，否则SpiderListener的onSuccess方法和onError方法都会执行
2016-03-12 20:03:27 +08:00
c0b8e8f8ae
remove .classpath .project
2016-01-22 14:58:22 +08:00
a8e6de4b90
Merge branch 'master' of git.oschina.net:flashsword20/webmagic
2016-01-22 10:16:58 +08:00
0fd4623f0a
Merge branch 'osc'
2016-01-21 19:33:30 +08:00
ce5495ecd5
remove useless files
2016-01-21 19:31:50 +08:00
8265c7dade
remove submodules for relase
2016-01-21 19:25:13 +08:00
7edfa26f90
complete javadoc
2016-01-21 18:34:07 +08:00
8b90b91e33
complete some javadoc
2016-01-21 18:14:10 +08:00
2b556cf053
update verison to 0.5.3-SNAPSHOT
2016-01-21 18:05:56 +08:00
9c5716a543
complete javadoc
2016-01-21 18:05:12 +08:00
db3cbf6ca5
update version to 0.5.3-SNAPSHOT
2016-01-21 17:58:36 +08:00
81ce1ffc5f
fix ignore
2016-01-21 12:36:49 +08:00
93764fa2c9
ignore some test
2016-01-21 12:28:32 +08:00
5706bb90af
update xsoup to 0.3.1
2016-01-20 12:59:11 +08:00
7586e3d75c
add some test for github repo downloader
2016-01-19 08:05:53 +08:00
90e14b31b0
修正FileCacheQueueScheduler导致程序不能正常结束和未关闭流
...
FileCacheQueueScheduler中开启了一个线程周期运行来保存数据但在爬虫结束后没有关闭导致程序无法结束，以及没有关闭io流。
解决方法：
让FileCacheQueueScheduler实现Closable接口，在close方法中关闭线程以及流。
在Spider的close方法中添加对scheduler的关闭操作。
2015-11-12 23:10:20 +08:00
56e0cd513a
compile error fix
2015-04-15 23:21:06 +08:00
c5740b1840
change assert #200
2015-04-15 08:32:08 +08:00
67eb632f4d
test for issue #200
2015-04-15 08:31:45 +08:00
590561a6e4
修正site.setHttpProxy()不起作用的bug
2015-03-09 15:54:15 +08:00
19474e4716
add SimpleProxyPool and IProxyPool
2015-02-28 17:50:10 +08:00
4978665633
add retry sleep time
2015-01-21 13:30:02 +08:00
8ffc1a7093
add NPE check for POST method
2015-01-13 14:10:00 +08:00
bc666e927d
Update Site.java
...
setCycleRetryTimes的javadoc是这么说的:Set cycleRetryTimes times when download fail, 0 by default. Only work in RedisScheduler.
而通过查看源码发现似乎并没有做限制,即只能用于RedisScheduler. 故想问一下该javadoc是否过时了?
2014-09-12 12:42:57 +08:00
147401ce5e
remove duplicate setPath in ProxyPool
2014-09-09 22:58:44 +08:00
e7668e01b8
fix SourceRegion error and add some tests on it #144
2014-08-21 14:29:06 +08:00
4446669c24
fix test
2014-08-18 10:54:24 +08:00
9866297ec4
Disable jsoup entity escape by Default. Set Html.DISABLE_HTML_ENTITY_ESCAPE to false to enable it. #149
2014-08-14 08:04:56 +08:00
4e6e946dd7
more friendly exception message in PlainText #144
2014-08-13 10:02:16 +08:00
yihua.huang
zhuyue
GZhY
xbynet
wuyifan
mei
Ckex.zha
Jerry
Jon
hepan
Linker Lin
zhangheng09
x1ny
高军
edwardsbean
zhugw