webMagic-Mobai
/
magic-Dependency
mirror of https://gitea.qinmian.online/Mobai/magic-Dependency.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
1,130 Commits
1 Branch
0 Tags
14 MiB
Commit Graph

547 Commits (34da2fb3a02708b562ec747679ef0cd8d171a042)

Author SHA1 Message Date
Sutra Zhou 34da2fb3a0 Make PageProcessor#getSite be default method. Closes #1040. 2021-10-24 23:20:38 +08:00
Sutra Zhou c5a037a807 Update for next development version 2021-07-22 13:02:46 +08:00
Sutra Zhou 4e51a4f68b Update versions for release 2021-07-22 12:42:55 +08:00
Sutra Zhou be6f5ff771 Add missing @Deprecated annotations. 2021-03-22 18:21:59 +08:00
Sutra Zhou 4e8a086dae Pass exception to onError. Fixes #1005. 2021-03-22 18:21:10 +08:00
Sutra Zhou d0e2776991 Upgrade xsoup from 0.3.1 to 0.3.2. 2021-01-10 14:10:32 +08:00
Sutra Zhou 0e01550a79 Upgrade dependencies, including the jedis from 2.9.3 to 3.4.1. 2021-01-06 03:21:10 +08:00
Sutra Zhou ab6ff7f809 Revert "pageCount修改" 
This reverts commit 9a71f0ac92.
 2021-01-02 20:33:32 +08:00
Sutra Zhou 30daec4803 Revert "提供出现某种异常刷新代理，异常可配置" 
This reverts commit 4a6441e7c5.
 2021-01-02 20:33:17 +08:00
Sutra Zhou d0843bee0d Revert "简化代码" 
This reverts commit 9cc5287743.
 2021-01-02 20:32:35 +08:00
Sutra Zhou 5ceccc62e0 Revert "提供异常刷新httpClient，异常可配置，重写getHttpClient代码" 
This reverts commit 19465089c3.
 2021-01-02 20:31:53 +08:00
Sutra Zhou 33e3fcdf22 Revert "代理接口的修改，提供刷星代理API。downloader 下载错误时，提供request,exception,proxyProvider三个参数，" 
This reverts commit ba69eba669.
 2021-01-02 20:27:28 +08:00
Sutra Zhou c489647c4b Revert " Downloader 提供刷新组件的api,方便在spider中操作" 
This reverts commit 2e2a0fdf3e.
 2021-01-02 20:15:10 +08:00
Sutra Zhou 4bedd97267 Revert " 刷新代理api重构，需要提供旧代理，如果依然是旧代理，才进行刷新，防止应延迟响应造成的过度刷新" 
This reverts commit 0aa2c3949d.
 2021-01-02 20:14:02 +08:00
Sutra Zhou 3f756c9325 Revert " 代理功能扩展，对原代理提供商进行拆分，加入lombok" 
This reverts commit 33906e36f4.
 2021-01-02 20:14:01 +08:00
Sutra Zhou aabc5584b8 Revert " bug修改，对结果提供缓存能力" 
This reverts commit f68795d7dd.
 2021-01-02 20:13:53 +08:00
Sutra Zhou 328c3e0d7d Remove useless imports to fix build. 2021-01-02 19:41:05 +08:00
Sutra Zhou 1d536cf705
Merge pull request #976 from yaoqiangpersonal/master 
主要是对代理的功能进行了增加和修改
 2020-12-29 17:27:42 +08:00
yao f68795d7dd bug修改，对结果提供缓存能力 2020-12-29 16:54:38 +08:00
yao 33906e36f4 代理功能扩展，对原代理提供商进行拆分，加入lombok 2020-12-29 16:18:43 +08:00
yao 0aa2c3949d 刷新代理api重构，需要提供旧代理，如果依然是旧代理，才进行刷新，防止应延迟响应造成的过度刷新 2020-12-22 18:19:37 +08:00
yao 2e2a0fdf3e Downloader 提供刷新组件的api,方便在spider中操作 2020-12-21 18:08:55 +08:00
yao 19465089c3 提供异常刷新httpClient，异常可配置，重写getHttpClient代码 2020-12-21 16:02:35 +08:00
yao 9cc5287743 简化代码 2020-12-21 14:58:01 +08:00
yao 4a6441e7c5 提供出现某种异常刷新代理，异常可配置 2020-12-21 14:52:25 +08:00
yao ba69eba669 代理接口的修改，提供刷星代理API。downloader 下载错误时，提供request,exception,proxyProvider三个参数， 2020-12-21 14:36:44 +08:00
itranlin fc7ae9ce28 子任务可以使用不同的下载器。。。 2020-12-19 17:59:52 +08:00
yao 9a71f0ac92 pageCount修改 2020-12-15 17:05:16 +08:00
Sutra Zhou 4b902270b4 Bump version number from 0.7.3 to 0.7.4. 2020-10-27 09:01:21 +08:00
Sutra Zhou b4b1df85a0 Fix TLSv1.3. Maybe we should expose a API to allow user to use org.apache.http.ssl.SSLContextBuilder. Fixes #948. 2020-09-21 17:48:59 +08:00
leeyazhou 9aab25f339 build: manage plugin version & remove build WARNING 
## use the new dependency of commons-io

[WARNING] The artifact org.apache.commons:commons-io:jar:1.3.2 has been
relocated to commons-io:commons-io:jar:1.3.2

## manage plugin version of maven-jar-plugin and maven-deploy-plugin

[WARNING]
[WARNING] Some problems were encountered while building the effective
model for us.codecraft:webmagic-core:jar:0.7.3
[WARNING] 'build.plugins.plugin.version' for
org.apache.maven.plugins:maven-jar-plugin is missing. @
us.codecraft:webmagic-parent:0.7.3, /opt/code/git/webmagic/pom.xml, line
263, column 21
[WARNING]
[WARNING] Some problems were encountered while building the effective
model for us.codecraft:webmagic-extension:jar:0.7.3
[WARNING] 'build.plugins.plugin.version' for
org.apache.maven.plugins:maven-jar-plugin is missing. @
us.codecraft:webmagic-parent:0.7.3, /opt/code/git/webmagic/pom.xml, line
263, column 21
[WARNING]
[WARNING] Some problems were encountered while building the effective
model for us.codecraft:webmagic-scripts:jar:0.7.3
[WARNING] 'build.plugins.plugin.version' for
org.apache.maven.plugins:maven-jar-plugin is missing. @ line 61, column
21
[WARNING]
[WARNING] Some problems were encountered while building the effective
model for us.codecraft:webmagic-selenium:jar:0.7.3
[WARNING] 'build.plugins.plugin.version' for
org.apache.maven.plugins:maven-jar-plugin is missing. @
us.codecraft:webmagic-parent:0.7.3, /opt/code/git/webmagic/pom.xml, line
263, column 21
[WARNING] 'build.plugins.plugin.version' for
org.apache.maven.plugins:maven-deploy-plugin is missing. @ line 34,
column 12
[WARNING]
[WARNING] Some problems were encountered while building the effective
model for us.codecraft:webmagic-saxon:jar:0.7.3
[WARNING] 'build.plugins.plugin.version' for
org.apache.maven.plugins:maven-jar-plugin is missing. @
us.codecraft:webmagic-parent:0.7.3, /opt/code/git/webmagic/pom.xml, line
263, column 21
[WARNING] 'build.plugins.plugin.version' for
org.apache.maven.plugins:maven-deploy-plugin is missing. @ line 34,
column 21
[WARNING]
[WARNING] Some problems were encountered while building the effective
model for us.codecraft:webmagic-samples:jar:0.7.3
[WARNING] 'build.plugins.plugin.version' for
org.apache.maven.plugins:maven-jar-plugin is missing. @
us.codecraft:webmagic-parent:0.7.3, /opt/code/git/webmagic/pom.xml, line
263, column 21
[WARNING]
[WARNING] Some problems were encountered while building the effective
model for us.codecraft:webmagic-parent:pom:0.7.3
[WARNING] 'build.plugins.plugin.version' for
org.apache.maven.plugins:maven-jar-plugin is missing. @ line 263, column
21
[WARNING]
[WARNING] It is highly recommended to fix these problems because they
threaten the stability of your build.
[WARNING]
[WARNING] For this reason, future Maven versions might no longer support
building such malformed projects.
 2020-08-07 16:36:32 +08:00
Sutra Zhou 48bc73fbff New method Proxy#create. 2020-06-24 13:43:16 +08:00
Sutra Zhou 6d3f2d9b64 Wrap URISyntaxException as IllegalArgumentException for Proxy#toURI. 2020-06-24 13:24:45 +08:00
Sutra Zhou 236e5ade44 Update Proxy#toString(). 2020-06-17 11:19:37 +08:00
Sutra Zhou 791323a5b0 Add Proxy#scheme. 2020-06-16 14:45:29 +08:00
Sutra Zhou 2413366adb Format code, no actual code changed. 2020-06-15 20:01:14 +08:00
Sutra Zhou 7945c0612d Merge branch 'master' of github.com:code4craft/webmagic 2020-05-30 02:10:25 +08:00
Sutra Zhou 71aa04c89f Upgrade dependencies. 2020-05-30 02:10:01 +08:00
Sutra Zhou f24410d7bf
Merge pull request #885 from Diffblue-benchmarks/add-NumberUtils-tests 
Add unit tests for us.codecraft.webmagic.utils.NumberUtils
 2020-05-29 14:49:12 +08:00
Sutra Zhou ab5ac9d796 Fix test failure and javadoc failure. 2020-05-28 19:05:17 +08:00
Sutra Zhou 4078766d0e Change log level of supportedProtocols. 2020-05-28 14:30:14 +08:00
Sutra Zhou 3e42523141 TLSv1.3 requires Java 11 at least. Fixes #927. 2020-05-28 14:14:16 +08:00
Sutra Zhou 436af97346 Use spaces as indent. 2020-05-28 13:59:05 +08:00
Sutra Zhou ba1b4017a7 Mark slf4j-log4j12 as optional. 2020-05-21 19:59:29 +08:00
Sutra Zhou e7476cb8dc Make Request#getExtra be generic. 2020-04-19 22:44:06 +08:00
Sutra Zhou c46400d126 Fix javadoc of sleep time. Fixes #918. 2020-04-12 01:30:57 +08:00
Sutra Zhou fe3d52e2a4 Add TLSv1.3 support. 2020-04-11 18:00:04 +08:00
Thomas Perkins 2fd0e192fd Add unit tests for us.codecraft.webmagic.utils.NumberUtils 
These tests were written using Diffblue Cover.
 2019-07-30 09:58:54 +01:00
yihua.huang e5db538c19 #647 remove ThreadSafe annotation 2017-11-29 13:49:40 +08:00
yihua.huang b539522ca8 #701 support to tls1.2 2017-11-29 13:36:19 +08:00