Joe Zhou
|
64e6a9800a
|
Add dead-lock note for QueueScheduler.
|
2022-11-24 00:26:09 +08:00 |
|
Joe Zhou
|
e735e4e585
|
Log the remaining capacity.
|
2022-11-20 18:31:36 +08:00 |
|
Joe Zhou
|
4915431845
|
Revise logging level from warn to info, as we have passed the exception to onError. refs #1094
|
2022-11-15 22:48:02 +08:00 |
|
Joe Zhou
|
075b98291b
|
Return spider in setEmptySleepTime itself for chainning.
|
2022-11-12 11:06:08 +08:00 |
|
Joe Zhou
|
ac912e8f1f
|
Revise QueueScheduler to support capacity-restricted.
|
2022-11-12 10:17:36 +08:00 |
|
Sutra Zhou
|
d2b2eed9df
|
Pass the task to onSuccess & onError.
|
2022-10-19 22:15:41 +08:00 |
|
Sutra Zhou
|
5f80e02abd
|
Interrupt current thread.
|
2022-10-19 22:08:38 +08:00 |
|
Sutra Zhou
|
126c32ecd0
|
Fix compatible issue.
|
2022-10-16 07:21:35 +08:00 |
vio.ao
|
e7a7fbeeeb
|
Enhance Jsoup could parse tr td tag directly
|
2022-10-01 18:23:15 +08:00 |
|
vio.ao
|
7a62a6cb45
|
Revert "Revert "Common the downloader status process and pass error information when …""
This reverts commit acfbd7b883.
|
2022-10-01 17:33:11 +08:00 |
|
Sutra Zhou
|
acfbd7b883
|
Revert "Common the downloader status process and pass error information when …"
|
2022-10-01 10:37:09 +08:00 |
|
vio.ao
|
d01f26333b
|
Common the downloader status process and pass error information when onError
|
2022-10-01 00:21:17 +08:00 |
David Hsing
|
54da7af17e
|
change dependency versions into properties
change dependency versions into properties
update commons-collections from 3.x to 4.4
|
2022-05-03 17:42:42 +08:00 |
|
Sutra Zhou
|
692605bd75
|
Merge branch 'develop' of github.com:code4craft/webmagic into develop
|
2021-10-24 23:20:56 +08:00 |
|
Sutra Zhou
|
34da2fb3a0
|
Make PageProcessor#getSite be default method. Closes #1040.
|
2021-10-24 23:20:38 +08:00 |
carl.don:tjr
|
fcdb9074d6
|
格式化 Spider.run 代码
|
2021-08-04 18:23:04 +08:00 |
|
carl.don:tjr
|
ab5d81a6b6
|
perfect Spider.run to avoid some rare concurrent issue, change the Spider.emptySleepTime to long type
|
2021-08-04 17:17:22 +08:00 |
|
Sutra Zhou
|
be6f5ff771
|
Add missing @Deprecated annotations.
|
2021-03-22 18:21:59 +08:00 |
|
Sutra Zhou
|
4e8a086dae
|
Pass exception to onError. Fixes #1005.
|
2021-03-22 18:21:10 +08:00 |
|
Sutra Zhou
|
0e01550a79
|
Upgrade dependencies, including the jedis from 2.9.3 to 3.4.1.
|
2021-01-06 03:21:10 +08:00 |
|
Sutra Zhou
|
ab6ff7f809
|
Revert "pageCount修改"
This reverts commit 9a71f0ac92.
|
2021-01-02 20:33:32 +08:00 |
|
Sutra Zhou
|
30daec4803
|
Revert "提供出现某种异常刷新代理，异常可配置"
This reverts commit 4a6441e7c5.
|
2021-01-02 20:33:17 +08:00 |
|
Sutra Zhou
|
d0843bee0d
|
Revert "简化代码"
This reverts commit 9cc5287743.
|
2021-01-02 20:32:35 +08:00 |
|
Sutra Zhou
|
5ceccc62e0
|
Revert "提供异常刷新httpClient，异常可配置，重写getHttpClient代码"
This reverts commit 19465089c3.
|
2021-01-02 20:31:53 +08:00 |
|
Sutra Zhou
|
33e3fcdf22
|
Revert "代理接口的修改，提供刷星代理API。downloader 下载错误时，提供request,exception,proxyProvider三个参数，"
This reverts commit ba69eba669.
|
2021-01-02 20:27:28 +08:00 |
|
Sutra Zhou
|
c489647c4b
|
Revert " Downloader 提供刷新组件的api,方便在spider中操作"
This reverts commit 2e2a0fdf3e.
|
2021-01-02 20:15:10 +08:00 |
|
Sutra Zhou
|
4bedd97267
|
Revert " 刷新代理api重构，需要提供旧代理，如果依然是旧代理，才进行刷新，防止应延迟响应造成的过度刷新"
This reverts commit 0aa2c3949d.
|
2021-01-02 20:14:02 +08:00 |
|
Sutra Zhou
|
3f756c9325
|
Revert " 代理功能扩展，对原代理提供商进行拆分，加入lombok"
This reverts commit 33906e36f4.
|
2021-01-02 20:14:01 +08:00 |
|
Sutra Zhou
|
aabc5584b8
|
Revert " bug修改，对结果提供缓存能力"
This reverts commit f68795d7dd.
|
2021-01-02 20:13:53 +08:00 |
|
Sutra Zhou
|
328c3e0d7d
|
Remove useless imports to fix build.
|
2021-01-02 19:41:05 +08:00 |
|
Sutra Zhou
|
1d536cf705
|
Merge pull request #976 from yaoqiangpersonal/master
主要是对代理的功能进行了增加和修改
|
2020-12-29 17:27:42 +08:00 |
yao
|
f68795d7dd
|
bug修改，对结果提供缓存能力
|
2020-12-29 16:54:38 +08:00 |
|
yao
|
33906e36f4
|
代理功能扩展，对原代理提供商进行拆分，加入lombok
|
2020-12-29 16:18:43 +08:00 |
|
yao
|
0aa2c3949d
|
刷新代理api重构，需要提供旧代理，如果依然是旧代理，才进行刷新，防止应延迟响应造成的过度刷新
|
2020-12-22 18:19:37 +08:00 |
|
yao
|
2e2a0fdf3e
|
Downloader 提供刷新组件的api,方便在spider中操作
|
2020-12-21 18:08:55 +08:00 |
|
yao
|
19465089c3
|
提供异常刷新httpClient，异常可配置，重写getHttpClient代码
|
2020-12-21 16:02:35 +08:00 |
|
yao
|
9cc5287743
|
简化代码
|
2020-12-21 14:58:01 +08:00 |
|
yao
|
4a6441e7c5
|
提供出现某种异常刷新代理，异常可配置
|
2020-12-21 14:52:25 +08:00 |
|
yao
|
ba69eba669
|
代理接口的修改，提供刷星代理API。downloader 下载错误时，提供request,exception,proxyProvider三个参数，
|
2020-12-21 14:36:44 +08:00 |
itranlin
|
fc7ae9ce28
|
子任务可以使用不同的下载器。。。
|
2020-12-19 17:59:52 +08:00 |
|
yao
|
9a71f0ac92
|
pageCount修改
|
2020-12-15 17:05:16 +08:00 |
|
Sutra Zhou
|
b4b1df85a0
|
Fix TLSv1.3. Maybe we should expose a API to allow user to use org.apache.http.ssl.SSLContextBuilder. Fixes #948.
|
2020-09-21 17:48:59 +08:00 |
|
Sutra Zhou
|
48bc73fbff
|
New method Proxy#create.
|
2020-06-24 13:43:16 +08:00 |
|
Sutra Zhou
|
6d3f2d9b64
|
Wrap URISyntaxException as IllegalArgumentException for Proxy#toURI.
|
2020-06-24 13:24:45 +08:00 |
|
Sutra Zhou
|
236e5ade44
|
Update Proxy#toString().
|
2020-06-17 11:19:37 +08:00 |
|
Sutra Zhou
|
791323a5b0
|
Add Proxy#scheme.
|
2020-06-16 14:45:29 +08:00 |
|
Sutra Zhou
|
2413366adb
|
Format code, no actual code changed.
|
2020-06-15 20:01:14 +08:00 |
|
Sutra Zhou
|
f24410d7bf
|
Merge pull request #885 from Diffblue-benchmarks/add-NumberUtils-tests
Add unit tests for us.codecraft.webmagic.utils.NumberUtils
|
2020-05-29 14:49:12 +08:00 |
|
Sutra Zhou
|
ab5ac9d796
|
Fix test failure and javadoc failure.
|
2020-05-28 19:05:17 +08:00 |
|
Sutra Zhou
|
4078766d0e
|
Change log level of supportedProtocols.
|
2020-05-28 14:30:14 +08:00 |