9aab25f339
build: manage plugin version & remove build WARNING
...
## use the new dependency of commons-io
[WARNING] The artifact org.apache.commons:commons-io:jar:1.3.2 has been
relocated to commons-io:commons-io:jar:1.3.2
## manage plugin version of maven-jar-plugin and maven-deploy-plugin
[WARNING]
[WARNING] Some problems were encountered while building the effective
model for us.codecraft:webmagic-core:jar:0.7.3
[WARNING] 'build.plugins.plugin.version' for
org.apache.maven.plugins:maven-jar-plugin is missing. @
us.codecraft:webmagic-parent:0.7.3, /opt/code/git/webmagic/pom.xml, line
263, column 21
[WARNING]
[WARNING] Some problems were encountered while building the effective
model for us.codecraft:webmagic-extension:jar:0.7.3
[WARNING] 'build.plugins.plugin.version' for
org.apache.maven.plugins:maven-jar-plugin is missing. @
us.codecraft:webmagic-parent:0.7.3, /opt/code/git/webmagic/pom.xml, line
263, column 21
[WARNING]
[WARNING] Some problems were encountered while building the effective
model for us.codecraft:webmagic-scripts:jar:0.7.3
[WARNING] 'build.plugins.plugin.version' for
org.apache.maven.plugins:maven-jar-plugin is missing. @ line 61, column
21
[WARNING]
[WARNING] Some problems were encountered while building the effective
model for us.codecraft:webmagic-selenium:jar:0.7.3
[WARNING] 'build.plugins.plugin.version' for
org.apache.maven.plugins:maven-jar-plugin is missing. @
us.codecraft:webmagic-parent:0.7.3, /opt/code/git/webmagic/pom.xml, line
263, column 21
[WARNING] 'build.plugins.plugin.version' for
org.apache.maven.plugins:maven-deploy-plugin is missing. @ line 34,
column 12
[WARNING]
[WARNING] Some problems were encountered while building the effective
model for us.codecraft:webmagic-saxon:jar:0.7.3
[WARNING] 'build.plugins.plugin.version' for
org.apache.maven.plugins:maven-jar-plugin is missing. @
us.codecraft:webmagic-parent:0.7.3, /opt/code/git/webmagic/pom.xml, line
263, column 21
[WARNING] 'build.plugins.plugin.version' for
org.apache.maven.plugins:maven-deploy-plugin is missing. @ line 34,
column 21
[WARNING]
[WARNING] Some problems were encountered while building the effective
model for us.codecraft:webmagic-samples:jar:0.7.3
[WARNING] 'build.plugins.plugin.version' for
org.apache.maven.plugins:maven-jar-plugin is missing. @
us.codecraft:webmagic-parent:0.7.3, /opt/code/git/webmagic/pom.xml, line
263, column 21
[WARNING]
[WARNING] Some problems were encountered while building the effective
model for us.codecraft:webmagic-parent:pom:0.7.3
[WARNING] 'build.plugins.plugin.version' for
org.apache.maven.plugins:maven-jar-plugin is missing. @ line 263, column
21
[WARNING]
[WARNING] It is highly recommended to fix these problems because they
threaten the stability of your build.
[WARNING]
[WARNING] For this reason, future Maven versions might no longer support
building such malformed projects.
2020-08-07 16:36:32 +08:00
48bc73fbff
New method Proxy#create.
2020-06-24 13:43:16 +08:00
6d3f2d9b64
Wrap URISyntaxException as IllegalArgumentException for Proxy#toURI.
2020-06-24 13:24:45 +08:00
236e5ade44
Update Proxy#toString().
2020-06-17 11:19:37 +08:00
791323a5b0
Add Proxy#scheme.
2020-06-16 14:45:29 +08:00
2413366adb
Format code, no actual code changed.
2020-06-15 20:01:14 +08:00
7945c0612d
Merge branch 'master' of github.com:code4craft/webmagic
2020-05-30 02:10:25 +08:00
71aa04c89f
Upgrade dependencies.
2020-05-30 02:10:01 +08:00
f24410d7bf
Merge pull request #885 from Diffblue-benchmarks/add-NumberUtils-tests
...
Add unit tests for us.codecraft.webmagic.utils.NumberUtils
2020-05-29 14:49:12 +08:00
ab5ac9d796
Fix test failure and javadoc failure.
2020-05-28 19:05:17 +08:00
4078766d0e
Change log level of supportedProtocols.
2020-05-28 14:30:14 +08:00
3e42523141
TLSv1.3 requires Java 11 at least. Fixes #927 .
2020-05-28 14:14:16 +08:00
436af97346
Use spaces as indent.
2020-05-28 13:59:05 +08:00
ba1b4017a7
Mark slf4j-log4j12 as optional.
2020-05-21 19:59:29 +08:00
e7476cb8dc
Make Request#getExtra be generic.
2020-04-19 22:44:06 +08:00
c46400d126
Fix javadoc of sleep time. Fixes #918 .
2020-04-12 01:30:57 +08:00
fe3d52e2a4
Add TLSv1.3 support.
2020-04-11 18:00:04 +08:00
2fd0e192fd
Add unit tests for us.codecraft.webmagic.utils.NumberUtils
...
These tests were written using Diffblue Cover.
2019-07-30 09:58:54 +01:00
e5db538c19
#647 remove ThreadSafe annotation
2017-11-29 13:49:40 +08:00
b539522ca8
#701 support to tls1.2
2017-11-29 13:36:19 +08:00
e276b11ea9
vesion 0.7.3
2017-07-30 15:29:22 +08:00
a7e3090711
setBinaryContent, setCharset 方法返回this, 方便链式调用
2017-07-29 19:07:40 +08:00
6f5b9e448e
#627 set charset to request
2017-07-29 11:27:56 +08:00
32f1f2cf44
#613 add charset to page
2017-07-29 11:16:09 +08:00
65049baca4
#571 fix spell mistake
2017-07-29 11:06:17 +08:00
f375b9fdba
#629 fix ut
2017-07-25 18:27:10 +08:00
3266ea15ca
#629 correct illegal url in HttpUriRequestConverter
2017-07-22 17:40:43 +08:00
5daf92e8b2
#610 CASE_INSENSITIVE for charset detect in Content-Type
2017-07-22 17:27:36 +08:00
2183ba9b61
#571 add getBytes to Page
2017-07-22 12:11:00 +08:00
c3bdb20458
#631 remove IllegalArgumentException of HttpRequestBody.json and so on
2017-07-22 11:49:23 +08:00
f405e642c0
add default constructor for HttpRequestBody #609
2017-06-24 11:31:17 +08:00
eb376fca74
update jsoup to 1.10.3 #608
2017-06-24 11:29:35 +08:00
faca38d4ec
update jsonpath to 2.2.0 #606
2017-06-24 11:01:23 +08:00
1b6394bef9
version 0.7.2
2017-06-17 16:10:10 +08:00
3ee00015c2
change header from Authorization to Proxy-Authorization for Proxy Authorization #596
2017-06-17 16:08:39 +08:00
4111b07263
more error log on page code error #601
2017-06-17 10:36:41 +08:00
6bd1eed25e
fix duplicate call of onSuccess and onError #605
2017-06-17 10:26:55 +08:00
3a589d4ca6
HttpRequestBody implements Serializable #594
2017-06-08 15:59:46 +08:00
13cdf82695
update version to 0.7.2-SNAPSHOT
2017-06-05 16:55:59 +08:00
1e9187f24e
version 0.7.1
2017-06-04 07:52:54 +08:00
592fa2c0f1
add site header test
2017-06-03 11:56:18 +08:00
19d34dbb65
not add bracket to regex in RegexSelector #559
2017-06-03 11:30:07 +08:00
b6b991a09b
Merge pull request #556 from zhuyuesut/master
...
增加对零宽断言的支持
2017-06-03 11:10:56 +08:00
bb0eb69acf
update ZhihuPageProcessor example
2017-06-03 10:42:17 +08:00
2e35e149be
for 0.7.1
2017-05-29 14:41:49 +08:00
17d8bfa907
docs and pgp version
2017-05-29 09:36:42 +08:00
17478fcfc4
0.7.0 release
2017-05-29 09:30:56 +08:00
636359300f
add Site.disableCookieManagement #577
2017-05-29 08:29:53 +08:00
49de9374cd
new SimpleHttpClient #576
2017-05-27 17:30:19 +08:00
8999ea9320
add public constructor for SimpleProxyProvider
2017-05-27 16:09:02 +08:00
a8c2e6c729
alpha release
2017-05-20 12:51:16 +08:00
3c1338193b
for 0.7.0.alpha
2017-05-20 12:34:09 +08:00
e8abc28072
#552 add some log when crawler stop
2017-05-20 11:44:51 +08:00
9e1b7ed3f7
Update RegexSelector.java
2017-05-05 10:47:10 +08:00
c80f25edbd
Update RegexSelectorTest.java
...
简单的增加了一点测试
2017-05-03 18:33:23 +08:00
c3183252ac
Update RegexSelector.java
2017-05-03 18:24:19 +08:00
cbf80af5dd
test for SimpleProxyProvider #535
2017-04-16 10:50:27 +08:00
eb632a93d3
SimpleProxyProvider #535
2017-04-16 10:43:56 +08:00
d38d51dfcb
fix javadoc
2017-04-15 12:24:50 +08:00
5f34adf938
完善 LinksSelector.selectList 的测试用例
2017-04-09 21:29:01 +08:00
ce3f0ac239
删除 fixAllRelativeHrefs 并修复 SeleniumDownloader 对 fixAllRelativeHrefs 的依赖
2017-04-09 21:01:32 +08:00
bc6e81e00f
修复checkElementAndConvert方法注释中注释错误
2017-04-09 20:40:00 +08:00
4a2c0f4f97
add returnProxy for proxyProvider
2017-04-09 09:28:36 +08:00
1b04a7f2b3
#527 move logic check from downloaderto spider
2017-04-09 09:23:10 +08:00
0f4d6e8b12
#525 remove port in UrlUtils.getDomain()
2017-04-08 23:17:00 +08:00
a1ae632b62
test for request cookies and headers
2017-04-08 23:13:16 +08:00
db67db8103
#523 remove fixAllRelativeHrefs by default, get absolute urls for links()
2017-04-08 22:06:18 +08:00
abd020b45b
some comments
2017-04-08 20:16:17 +08:00
2622b448b8
fix test
2017-04-08 20:09:43 +08:00
b06a248c00
fix test
2017-04-08 20:06:04 +08:00
1cfbd13aae
refacor in httpclientdownloader
2017-04-08 20:04:56 +08:00
83ada9749e
fix test
2017-04-08 12:16:34 +08:00
fe95a6842f
Request再次重构:去掉params，仅保留HttpRequestBody
2017-04-08 12:12:39 +08:00
395396c68e
增加HttpRequestBody
2017-04-08 11:59:52 +08:00
c93a8a2722
修复字符编码检测BUG
2017-03-31 18:27:18 +08:00
74110e6ec5
remove useless file
2017-03-30 09:19:00 +08:00
b100dfe273
update version
2017-03-30 09:18:12 +08:00
1c24baa8d1
Request支持设置header与cookie
...
新增POST请求时,XML、JSON参数支持
Page支持获取响应header
2017-03-29 02:27:06 +08:00
6bd197859b
fix test
2017-03-27 09:55:50 +08:00
f23e138c72
add response headers to Page #508
2017-03-27 09:52:25 +08:00
c13110c4cb
fix samples
2017-03-21 07:53:43 +08:00
c51ac6017c
remove Site.addStartRequest() etc. #494
2017-03-21 07:32:34 +08:00
68050fc88e
test pass
2017-03-21 07:19:12 +08:00
474b7c9d57
refactor
2017-03-20 23:13:49 +08:00
25c81013ca
new proxy pool api
2017-03-20 22:47:24 +08:00
46297deaa1
HttpUriRequestConverter
2017-03-20 22:43:32 +08:00
1d86f7c048
compile passed in httpclientDownloader
2017-03-20 22:40:14 +08:00
b71f379512
fix
2017-03-18 12:18:00 +08:00
a7f9e7cad5
重构一部分httpclient
2017-03-18 12:16:21 +08:00
221c155060
move release connection before return proxy #396
2017-03-18 11:15:36 +08:00
68beff42c5
add test #493
2017-03-18 11:01:30 +08:00
79522f941e
Bug, add null check to site in HttpClientDownloader & HttpClientGenerator
2017-03-17 14:10:54 +08:00
e9341d0291
complete test #447
2017-03-17 07:54:28 +08:00
e7d35c4846
add params to all method of request #447
2017-03-17 07:18:05 +08:00
75bad591d7
rewrite hashCode and equals for params #447
2017-03-17 07:10:14 +08:00
11c32669b2
Merge pull request #447 from xbynet/master
...
简化POST参数设置.
2017-03-17 07:06:46 +08:00
aa01e27779
change constructor for Proxy to public #490
2017-03-17 07:02:02 +08:00
791520e6a0
fix a bug of RegexSelector when regex has zero-width assertions.
2017-03-17 00:06:15 +08:00
c175ea88c0
#more test #484
2017-03-11 11:43:18 +08:00
9b964c0a99
test for #484
2017-03-11 11:41:01 +08:00
fc702fd3b6
introduce mockito for test
2017-03-11 11:31:15 +08:00
5215a492cc
remove duplicate check for POST request #484
2017-03-11 11:26:13 +08:00
0a1fb19052
add tests #483
2017-03-11 10:56:31 +08:00
a2e7f0004b
Merge branch 'master' of github.com:code4craft/webmagic
2017-03-11 10:52:54 +08:00
ef32571821
rewrite Request.equals and hashCode, add Method to check #483
2017-03-11 10:52:39 +08:00
8b8f535c30
refactor:extract charset detect to utils
2017-03-11 10:43:10 +08:00
e645524ad2
fix bug,set ExecutorService
2017-03-04 20:57:29 +08:00
a872a6480e
fix code sample for github #348
2017-02-25 22:46:29 +08:00
1d2171805f
add test for #228
2017-02-25 22:30:48 +08:00
bbe0b52ddd
remove synchronized in QueueScheduler #410
2017-02-25 19:55:45 +08:00
ad69963005
remove synchronize in Page #411
2017-02-25 19:42:12 +08:00
3a796b9413
remove duplicate code #421
2017-02-25 12:01:12 +08:00
42f1018010
remove messy code
2017-02-21 14:08:05 +08:00
650468c0e4
解决POST中文参数乱码问题
2017-01-22 18:04:22 +08:00
aaccc93215
new version
2017-01-21 12:04:12 +08:00
3e633c6871
version
2017-01-21 11:51:14 +08:00
f45e2f118b
for release
2017-01-21 11:38:36 +08:00
d60615f503
修复使用startUrls没有设置domain导致使用cookie空指针的问题#438
2017-01-21 11:29:42 +08:00
407fbb6130
refactor logger#445
2017-01-21 11:05:54 +08:00
0dc26c8ca0
optimize code.
2017-01-20 14:03:26 +08:00
4f76d62d4f
Merge pull request #444 from ckex/develop
...
绕过安全证书
2017-01-18 23:42:51 +08:00
e4af05a6f2
绕过安全证书
2017-01-18 17:28:01 +08:00
c23627bf63
解决post/redirect/post 302跳转问题
2017-01-17 00:07:01 +08:00
d69204b919
0.6.0
2016-12-18 11:45:43 +08:00
9bdb48b2d0
version 0.6.0
2016-12-18 11:20:28 +08:00
eeb607fd0e
将Spider.processRequest()抛出异常改回原来的逻辑
2016-12-18 11:04:58 +08:00
97592d6720
Version 0.6.0
2016-12-18 10:58:24 +08:00
00dfebbceb
#424 remove guava dep and add fix docs
2016-12-18 10:45:50 +08:00
c2531c6817
clean dependency
2016-12-18 08:34:46 +08:00
a960a39c44
fix compile error for example change
2016-12-18 08:32:14 +08:00
7476ceccee
more stable test
2016-12-18 08:15:26 +08:00
5ce3fdfe5a
some refactor in log
2016-12-18 08:15:09 +08:00
98163a3e40
update examples
2016-12-18 07:46:18 +08:00
b090dcd20d
sepcific error page for HttpClientDownloaderTest to avoid test error when local port is available
2016-12-18 07:15:06 +08:00
8f942d6fe2
#419 修复抓取https链接线程无法结束导致进程一直运行的问题
2016-12-18 06:56:01 +08:00
dafd2b77ff
fix GithubRepoPageProcessor in example
2016-11-24 08:18:06 +08:00
cfed860fb9
Merge branch 'master' of github.com:code4craft/webmagic
2016-11-22 17:00:27 +08:00
2189aab652
fix test
2016-11-22 16:58:49 +08:00
1491033534
Merge pull request #377 from jerry-sc/monitor-bug
...
fix the monitor bug which the spider will terminate when a seed url with port
2016-11-19 13:01:30 +08:00
507556d0aa
fix test: ProxyTest.testProxy() do not load exist proxy config
2016-11-19 12:54:39 +08:00
e56b8c3efc
fix the monitor bug which the spider will terminate when a seed url with port
2016-09-22 22:36:18 +08:00
448e528140
update StringUtils to apache lang3 #314
2016-05-24 13:33:17 +08:00
3e33959b7a
#319 fix javadoc
2016-05-24 13:17:35 +08:00
8730e3e97a
Merge branch 'fix' of git://github.com/kapsterio/webmagic into kapsterio-fix
2016-05-08 20:46:22 +08:00
2400ff7e1a
resovle conflict
2016-05-08 20:31:43 +08:00
b7f3c4bba0
Merge branch 'master' of git://github.com/hepan/webmagic into hepan-master
2016-05-08 20:27:47 +08:00
d8f978fd20
fix test in JsonPathSelectorTest #289
2016-05-08 19:32:03 +08:00
61c28a0130
refactor on proxypool
2016-05-08 17:53:15 +08:00
b871b210c5
Merge branch 'proxy-strategy' of github.com:EdwardsBean/webmagic into EdwardsBean-proxy-strategy
2016-05-08 17:53:02 +08:00
b5413368de
update ut
2016-05-08 16:23:41 +08:00
leeyazhou
Sutra Zhou
Thomas Perkins
yihua.huang
王侃
zhuyue
GZhY
xbynet
wuyifan
mei
Ckex.zha
Jerry