Mobai
/
magic-Dependency
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
1,292 Commits
1 Branch
0 Tags
14 MiB
Commit Graph

1292 Commits (master)

Author SHA1 Message Date
Sutra Zhou 328c3e0d7d Remove useless imports to fix build. 2021-01-02 19:41:05 +08:00
Sutra Zhou 1d536cf705
Merge pull request #976 from yaoqiangpersonal/master 
主要是对代理的功能进行了增加和修改
 2020-12-29 17:27:42 +08:00
yao f68795d7dd bug修改，对结果提供缓存能力 2020-12-29 16:54:38 +08:00
yao 33906e36f4 代理功能扩展，对原代理提供商进行拆分，加入lombok 2020-12-29 16:18:43 +08:00
yao 0aa2c3949d 刷新代理api重构，需要提供旧代理，如果依然是旧代理，才进行刷新，防止应延迟响应造成的过度刷新 2020-12-22 18:19:37 +08:00
yao 2e2a0fdf3e Downloader 提供刷新组件的api,方便在spider中操作 2020-12-21 18:08:55 +08:00
yao 19465089c3 提供异常刷新httpClient，异常可配置，重写getHttpClient代码 2020-12-21 16:02:35 +08:00
yao 9cc5287743 简化代码 2020-12-21 14:58:01 +08:00
yao 4a6441e7c5 提供出现某种异常刷新代理，异常可配置 2020-12-21 14:52:25 +08:00
yao ba69eba669 代理接口的修改，提供刷星代理API。downloader 下载错误时，提供request,exception,proxyProvider三个参数， 2020-12-21 14:36:44 +08:00
Sutra Zhou b2aa5e2677
Merge pull request #974 from itranlin/master 
子任务可以使用不同的下载器
 2020-12-19 18:12:17 +08:00
itranlin fc7ae9ce28 子任务可以使用不同的下载器。。。 2020-12-19 17:59:52 +08:00
yao 9a71f0ac92 pageCount修改 2020-12-15 17:05:16 +08:00
Sutra Zhou 4b902270b4 Bump version number from 0.7.3 to 0.7.4. 2020-10-27 09:01:21 +08:00
Sutra Zhou 1895dba696
Merge pull request #959 from code4craft/snyk-fix-ad3d8147b4d9640ed91bfc34fbfb9e38 
[Snyk] Security upgrade com.google.guava:guava from 29.0-jre to 30.0-android
 2020-10-24 15:52:10 +08:00
snyk-bot c87af365d6
fix: pom.xml to reduce vulnerabilities 
The following vulnerabilities are fixed with an upgrade:
- https://snyk.io/vuln/SNYK-JAVA-COMGOOGLEGUAVA-1015415
 2020-10-24 03:02:20 +00:00
Sutra Zhou 541f421011
Merge pull request #957 from code4craft/snyk-fix-9aefd7a47e1f776c68e7fad3e85033a4 
[Snyk] Security upgrade junit:junit from 4.13 to 4.13.1
 2020-10-14 10:23:49 +08:00
snyk-bot 2223552aeb
fix: pom.xml to reduce vulnerabilities 
The following vulnerabilities are fixed with an upgrade:
- https://snyk.io/vuln/SNYK-JAVA-JUNIT-1017047
 2020-10-14 00:46:32 +00:00
Sutra Zhou 50c4e8ccfe
Merge pull request #955 from code4craft/snyk-fix-5cfb952010fd88492e4e2d5516c6237d 
[Snyk] Security upgrade org.apache.httpcomponents:httpclient from 4.5.12 to 4.5.13
 2020-10-10 11:00:09 +08:00
snyk-bot 7f737626b1
fix: pom.xml to reduce vulnerabilities 
The following vulnerabilities are fixed with an upgrade:
- https://snyk.io/vuln/SNYK-JAVA-ORGAPACHEHTTPCOMPONENTS-1016906
 2020-10-10 02:58:58 +00:00
Sutra Zhou b4b1df85a0 Fix TLSv1.3. Maybe we should expose a API to allow user to use org.apache.http.ssl.SSLContextBuilder. Fixes #948. 2020-09-21 17:48:59 +08:00
Sutra Zhou 94ac7ca3b6
Merge pull request #946 from code4craft/snyk-fix-84ae966502c97386fb9c384e5874967b 
[Snyk] Security upgrade com.alibaba:fastjson from 1.2.68 to 1.2.69
 2020-09-09 11:02:19 +08:00
snyk-bot e3b3b9afdd
fix: pom.xml to reduce vulnerabilities 
The following vulnerabilities are fixed with an upgrade:
- https://snyk.io/vuln/SNYK-JAVA-COMALIBABA-570967
 2020-09-09 02:59:41 +00:00
Sutra Zhou 96ebe608cc
Merge pull request #939 from bytesgo/manage-version 
build: manage plugin version & remove build WARNING
 2020-08-07 17:38:17 +08:00
leeyazhou 9aab25f339 build: manage plugin version & remove build WARNING 
## use the new dependency of commons-io

[WARNING] The artifact org.apache.commons:commons-io:jar:1.3.2 has been
relocated to commons-io:commons-io:jar:1.3.2

## manage plugin version of maven-jar-plugin and maven-deploy-plugin

[WARNING]
[WARNING] Some problems were encountered while building the effective
model for us.codecraft:webmagic-core:jar:0.7.3
[WARNING] 'build.plugins.plugin.version' for
org.apache.maven.plugins:maven-jar-plugin is missing. @
us.codecraft:webmagic-parent:0.7.3, /opt/code/git/webmagic/pom.xml, line
263, column 21
[WARNING]
[WARNING] Some problems were encountered while building the effective
model for us.codecraft:webmagic-extension:jar:0.7.3
[WARNING] 'build.plugins.plugin.version' for
org.apache.maven.plugins:maven-jar-plugin is missing. @
us.codecraft:webmagic-parent:0.7.3, /opt/code/git/webmagic/pom.xml, line
263, column 21
[WARNING]
[WARNING] Some problems were encountered while building the effective
model for us.codecraft:webmagic-scripts:jar:0.7.3
[WARNING] 'build.plugins.plugin.version' for
org.apache.maven.plugins:maven-jar-plugin is missing. @ line 61, column
21
[WARNING]
[WARNING] Some problems were encountered while building the effective
model for us.codecraft:webmagic-selenium:jar:0.7.3
[WARNING] 'build.plugins.plugin.version' for
org.apache.maven.plugins:maven-jar-plugin is missing. @
us.codecraft:webmagic-parent:0.7.3, /opt/code/git/webmagic/pom.xml, line
263, column 21
[WARNING] 'build.plugins.plugin.version' for
org.apache.maven.plugins:maven-deploy-plugin is missing. @ line 34,
column 12
[WARNING]
[WARNING] Some problems were encountered while building the effective
model for us.codecraft:webmagic-saxon:jar:0.7.3
[WARNING] 'build.plugins.plugin.version' for
org.apache.maven.plugins:maven-jar-plugin is missing. @
us.codecraft:webmagic-parent:0.7.3, /opt/code/git/webmagic/pom.xml, line
263, column 21
[WARNING] 'build.plugins.plugin.version' for
org.apache.maven.plugins:maven-deploy-plugin is missing. @ line 34,
column 21
[WARNING]
[WARNING] Some problems were encountered while building the effective
model for us.codecraft:webmagic-samples:jar:0.7.3
[WARNING] 'build.plugins.plugin.version' for
org.apache.maven.plugins:maven-jar-plugin is missing. @
us.codecraft:webmagic-parent:0.7.3, /opt/code/git/webmagic/pom.xml, line
263, column 21
[WARNING]
[WARNING] Some problems were encountered while building the effective
model for us.codecraft:webmagic-parent:pom:0.7.3
[WARNING] 'build.plugins.plugin.version' for
org.apache.maven.plugins:maven-jar-plugin is missing. @ line 263, column
21
[WARNING]
[WARNING] It is highly recommended to fix these problems because they
threaten the stability of your build.
[WARNING]
[WARNING] For this reason, future Maven versions might no longer support
building such malformed projects.
 2020-08-07 16:36:32 +08:00
Sutra Zhou 48bc73fbff New method Proxy#create. 2020-06-24 13:43:16 +08:00
Sutra Zhou 6d3f2d9b64 Wrap URISyntaxException as IllegalArgumentException for Proxy#toURI. 2020-06-24 13:24:45 +08:00
Sutra Zhou 236e5ade44 Update Proxy#toString(). 2020-06-17 11:19:37 +08:00
Sutra Zhou 791323a5b0 Add Proxy#scheme. 2020-06-16 14:45:29 +08:00
Sutra Zhou 2413366adb Format code, no actual code changed. 2020-06-15 20:01:14 +08:00
Sutra Zhou 5d14efc50f Serialize request URL only in FileCacheQueueScheduler. 2020-06-14 00:20:39 +08:00
Sutra Zhou 7945c0612d Merge branch 'master' of github.com:code4craft/webmagic 2020-05-30 02:10:25 +08:00
Sutra Zhou 71aa04c89f Upgrade dependencies. 2020-05-30 02:10:01 +08:00
Sutra Zhou 6c05dd8b72 Upgrade maven plugins. 2020-05-29 18:19:45 +08:00
Sutra Zhou a7c4e701e4 Specify the required minimum maven version. 2020-05-29 18:12:05 +08:00
Sutra Zhou f24410d7bf
Merge pull request #885 from Diffblue-benchmarks/add-NumberUtils-tests 
Add unit tests for us.codecraft.webmagic.utils.NumberUtils
 2020-05-29 14:49:12 +08:00
Sutra Zhou c966a2015c
Merge pull request #882 from code4craft/snyk-fix-cec34e294bd875db1e52fde5b4b853f1 
[Snyk] Fix for 1 vulnerable dependencies
 2020-05-29 14:45:08 +08:00
Sutra Zhou 98281ab26e Upgrade httpclient from 4.5.2 to 4.5.12. 2020-05-29 14:43:59 +08:00
Sutra Zhou 0af0badd25
Merge pull request #889 from code4craft/snyk-fix-e3a42154e495a09f41dc3b2a6e6c9327 
[Snyk] Fix for 15 vulnerable dependencies
 2020-05-29 14:42:13 +08:00
Sutra Zhou c97dcfd93b
Merge pull request #925 from code4craft/snyk-fix-f95684b6c6d9efe03af61ca1db60f547 
[Snyk] Security upgrade com.github.dreamhead:moco-core from 0.11.0 to 1.0.0
 2020-05-29 14:41:41 +08:00
Sutra Zhou e0d98df20c
Merge pull request #923 from code4craft/snyk-fix-a77ec7619ec5042ae3cc039eeb4b728b 
[Snyk] Security upgrade com.github.dreamhead:moco-core from 0.11.0 to 1.0.0
 2020-05-29 14:40:57 +08:00
Sutra Zhou 750d656ae9
Merge pull request #922 from code4craft/snyk-fix-c555226a9da1135c3953aafa295a803d 
[Snyk] Security upgrade com.github.dreamhead:moco-core from 0.11.0 to 1.0.0
 2020-05-29 14:38:54 +08:00
Sutra Zhou 06386a72ce
Merge pull request #920 from code4craft/snyk-fix-7d7c3e630152d251e6b5661ebcb7903f 
[Snyk] Security upgrade com.github.dreamhead:moco-core from 0.11.0 to 1.0.0
 2020-05-29 14:38:31 +08:00
Sutra Zhou 97738aee46
Merge pull request #914 from code4craft/snyk-fix-991184deb41d150aace33c2c66628d25 
[Snyk] Security upgrade com.github.dreamhead:moco-core from 0.11.0 to 1.0.0
 2020-05-29 14:38:08 +08:00
Sutra Zhou d1cb985b57
Merge pull request #911 from code4craft/snyk-fix-3eaeb3b8db2efeeb547c4607ef121a7c 
[Snyk] Fix for 1 vulnerabilities
 2020-05-29 14:37:45 +08:00
Sutra Zhou 155825088c
Merge pull request #908 from code4craft/snyk-fix-136a1b205198304fbf279cba094ed737 
[Snyk] Fix for 1 vulnerabilities
 2020-05-29 14:37:24 +08:00
Sutra Zhou 410daa94e1
Merge pull request #899 from code4craft/snyk-fix-858549a4b6d8c4cb61600052a08ab66d 
[Snyk] Fix for 1 vulnerabilities
 2020-05-29 14:37:00 +08:00
Sutra Zhou 86cc3c8659
Merge pull request #897 from code4craft/snyk-fix-e41912d25ca79fe77c7dc4748b211c9a 
[Snyk] Fix for 1 vulnerabilities
 2020-05-29 14:36:21 +08:00
Sutra Zhou fe9dca1247 Upgrade guava from 15.0 to 29.0. 2020-05-29 14:28:34 +08:00
Sutra Zhou d7f8c1d862
Merge pull request #895 from code4craft/snyk-fix-ac48823ab9abac16eb16ea3ff73b9be2 
[Snyk] Fix for 2 vulnerabilities
 2020-05-29 14:01:08 +08:00