Sutra Zhou
|
ab6ff7f809
|
Revert "pageCount修改"
This reverts commit 9a71f0ac92.
|
2021-01-02 20:33:32 +08:00 |
|
Sutra Zhou
|
30daec4803
|
Revert "提供出现某种异常刷新代理，异常可配置"
This reverts commit 4a6441e7c5.
|
2021-01-02 20:33:17 +08:00 |
|
Sutra Zhou
|
d0843bee0d
|
Revert "简化代码"
This reverts commit 9cc5287743.
|
2021-01-02 20:32:35 +08:00 |
|
Sutra Zhou
|
5ceccc62e0
|
Revert "提供异常刷新httpClient，异常可配置，重写getHttpClient代码"
This reverts commit 19465089c3.
|
2021-01-02 20:31:53 +08:00 |
|
Sutra Zhou
|
33e3fcdf22
|
Revert "代理接口的修改，提供刷星代理API。downloader 下载错误时，提供request,exception,proxyProvider三个参数，"
This reverts commit ba69eba669.
|
2021-01-02 20:27:28 +08:00 |
|
Sutra Zhou
|
c489647c4b
|
Revert " Downloader 提供刷新组件的api,方便在spider中操作"
This reverts commit 2e2a0fdf3e.
|
2021-01-02 20:15:10 +08:00 |
|
Sutra Zhou
|
4bedd97267
|
Revert " 刷新代理api重构，需要提供旧代理，如果依然是旧代理，才进行刷新，防止应延迟响应造成的过度刷新"
This reverts commit 0aa2c3949d.
|
2021-01-02 20:14:02 +08:00 |
|
Sutra Zhou
|
3f756c9325
|
Revert " 代理功能扩展，对原代理提供商进行拆分，加入lombok"
This reverts commit 33906e36f4.
|
2021-01-02 20:14:01 +08:00 |
|
Sutra Zhou
|
aabc5584b8
|
Revert " bug修改，对结果提供缓存能力"
This reverts commit f68795d7dd.
|
2021-01-02 20:13:53 +08:00 |
|
Sutra Zhou
|
328c3e0d7d
|
Remove useless imports to fix build.
|
2021-01-02 19:41:05 +08:00 |
|
Sutra Zhou
|
1d536cf705
|
Merge pull request #976 from yaoqiangpersonal/master
主要是对代理的功能进行了增加和修改
|
2020-12-29 17:27:42 +08:00 |
yao
|
f68795d7dd
|
bug修改，对结果提供缓存能力
|
2020-12-29 16:54:38 +08:00 |
|
yao
|
33906e36f4
|
代理功能扩展，对原代理提供商进行拆分，加入lombok
|
2020-12-29 16:18:43 +08:00 |
|
yao
|
0aa2c3949d
|
刷新代理api重构，需要提供旧代理，如果依然是旧代理，才进行刷新，防止应延迟响应造成的过度刷新
|
2020-12-22 18:19:37 +08:00 |
|
yao
|
2e2a0fdf3e
|
Downloader 提供刷新组件的api,方便在spider中操作
|
2020-12-21 18:08:55 +08:00 |
|
yao
|
19465089c3
|
提供异常刷新httpClient，异常可配置，重写getHttpClient代码
|
2020-12-21 16:02:35 +08:00 |
|
yao
|
9cc5287743
|
简化代码
|
2020-12-21 14:58:01 +08:00 |
|
yao
|
4a6441e7c5
|
提供出现某种异常刷新代理，异常可配置
|
2020-12-21 14:52:25 +08:00 |
|
yao
|
ba69eba669
|
代理接口的修改，提供刷星代理API。downloader 下载错误时，提供request,exception,proxyProvider三个参数，
|
2020-12-21 14:36:44 +08:00 |
itranlin
|
fc7ae9ce28
|
子任务可以使用不同的下载器。。。
|
2020-12-19 17:59:52 +08:00 |
|
yao
|
9a71f0ac92
|
pageCount修改
|
2020-12-15 17:05:16 +08:00 |
|
Sutra Zhou
|
b4b1df85a0
|
Fix TLSv1.3. Maybe we should expose a API to allow user to use org.apache.http.ssl.SSLContextBuilder. Fixes #948.
|
2020-09-21 17:48:59 +08:00 |
|
Sutra Zhou
|
48bc73fbff
|
New method Proxy#create.
|
2020-06-24 13:43:16 +08:00 |
|
Sutra Zhou
|
6d3f2d9b64
|
Wrap URISyntaxException as IllegalArgumentException for Proxy#toURI.
|
2020-06-24 13:24:45 +08:00 |
|
Sutra Zhou
|
236e5ade44
|
Update Proxy#toString().
|
2020-06-17 11:19:37 +08:00 |
|
Sutra Zhou
|
791323a5b0
|
Add Proxy#scheme.
|
2020-06-16 14:45:29 +08:00 |
|
Sutra Zhou
|
2413366adb
|
Format code, no actual code changed.
|
2020-06-15 20:01:14 +08:00 |
|
Sutra Zhou
|
f24410d7bf
|
Merge pull request #885 from Diffblue-benchmarks/add-NumberUtils-tests
Add unit tests for us.codecraft.webmagic.utils.NumberUtils
|
2020-05-29 14:49:12 +08:00 |
|
Sutra Zhou
|
ab5ac9d796
|
Fix test failure and javadoc failure.
|
2020-05-28 19:05:17 +08:00 |
|
Sutra Zhou
|
4078766d0e
|
Change log level of supportedProtocols.
|
2020-05-28 14:30:14 +08:00 |
|
Sutra Zhou
|
3e42523141
|
TLSv1.3 requires Java 11 at least. Fixes #927.
|
2020-05-28 14:14:16 +08:00 |
|
Sutra Zhou
|
436af97346
|
Use spaces as indent.
|
2020-05-28 13:59:05 +08:00 |
|
Sutra Zhou
|
e7476cb8dc
|
Make Request#getExtra be generic.
|
2020-04-19 22:44:06 +08:00 |
|
Sutra Zhou
|
c46400d126
|
Fix javadoc of sleep time. Fixes #918.
|
2020-04-12 01:30:57 +08:00 |
|
Sutra Zhou
|
fe3d52e2a4
|
Add TLSv1.3 support.
|
2020-04-11 18:00:04 +08:00 |
Thomas Perkins
|
2fd0e192fd
|
Add unit tests for us.codecraft.webmagic.utils.NumberUtils
These tests were written using Diffblue Cover.
|
2019-07-30 09:58:54 +01:00 |
yihua.huang
|
e5db538c19
|
#647 remove ThreadSafe annotation
|
2017-11-29 13:49:40 +08:00 |
|
yihua.huang
|
b539522ca8
|
#701 support to tls1.2
|
2017-11-29 13:36:19 +08:00 |
王侃
|
a7e3090711
|
setBinaryContent, setCharset 方法返回this, 方便链式调用
|
2017-07-29 19:07:40 +08:00 |
|
yihua.huang
|
6f5b9e448e
|
#627 set charset to request
|
2017-07-29 11:27:56 +08:00 |
|
yihua.huang
|
32f1f2cf44
|
#613 add charset to page
|
2017-07-29 11:16:09 +08:00 |
|
yihua.huang
|
65049baca4
|
#571 fix spell mistake
|
2017-07-29 11:06:17 +08:00 |
|
yihua.huang
|
f375b9fdba
|
#629 fix ut
|
2017-07-25 18:27:10 +08:00 |
|
yihua.huang
|
3266ea15ca
|
#629 correct illegal url in HttpUriRequestConverter
|
2017-07-22 17:40:43 +08:00 |
|
yihua.huang
|
5daf92e8b2
|
#610 CASE_INSENSITIVE for charset detect in Content-Type
|
2017-07-22 17:27:36 +08:00 |
|
yihua.huang
|
2183ba9b61
|
#571 add getBytes to Page
|
2017-07-22 12:11:00 +08:00 |
|
yihua.huang
|
c3bdb20458
|
#631 remove IllegalArgumentException of HttpRequestBody.json and so on
|
2017-07-22 11:49:23 +08:00 |
|
yihua.huang
|
f405e642c0
|
add default constructor for HttpRequestBody #609
|
2017-06-24 11:31:17 +08:00 |
|
yihua.huang
|
eb376fca74
|
update jsoup to 1.10.3 #608
|
2017-06-24 11:29:35 +08:00 |
|
yihua.huang
|
faca38d4ec
|
update jsonpath to 2.2.0 #606
|
2017-06-24 11:01:23 +08:00 |