Sutra Zhou
|
a0ff4a2d2e
|
Fix log message.
|
2024-04-23 20:09:15 +08:00 |
Harikrishna
|
7c20290ce4
|
Refactor addTargetRequests method to eliminate redundant code. (#1138)
|
2023-11-26 10:56:06 +08:00 |
|
Harikrishna
|
a911104076
|
Refactored to remove multiple calls of getSourceTexts() api (#1137)
|
2023-11-24 20:09:32 +08:00 |
|
Harikrishna
|
622ed5a17f
|
Refactor compareLong method using Long.compare, corrected the local variable name (#1136)
|
2023-11-24 12:37:04 +08:00 |
|
Sutra Zhou
|
67644de3d9
|
Expose Page to onSuccess & onError.
|
2023-11-20 18:28:27 +08:00 |
|
Joe Zhou
|
eda3be9432
|
Fix log format.
|
2023-11-06 22:44:53 +08:00 |
|
Joe Zhou
|
ad010927f6
|
Fix test.
|
2023-06-22 10:40:46 +08:00 |
|
Joe Zhou
|
58fd08bcf8
|
Expose Request to ProxyProvider.
|
2023-06-22 09:52:58 +08:00 |
Tanky-Zhang
|
244ade7b4c
|
feat:update host verify (#1112)
|
2023-03-22 22:25:51 +08:00 |
hooyantsing
|
12ce86425f
|
BugFix: Jsoup 和 HtmlCleaner 构建 Dom 时，若缺失 table 标签，则无法正常解析 tr 和 td 标签。
|
2023-02-03 22:48:58 +08:00 |
|
Joe Zhou
|
a266df406f
|
Add Site.defaultCharset. closes #1101.
|
2022-12-20 23:41:31 +08:00 |
|
Joe Zhou
|
80424b0bd7
|
Replace List with Iterable, fixed #1099.
|
2022-12-05 23:26:01 +08:00 |
|
Joe Zhou
|
64e6a9800a
|
Add dead-lock note for QueueScheduler.
|
2022-11-24 00:26:09 +08:00 |
|
Joe Zhou
|
e735e4e585
|
Log the remaining capacity.
|
2022-11-20 18:31:36 +08:00 |
|
Joe Zhou
|
4915431845
|
Revise logging level from warn to info, as we have passed the exception to onError. refs #1094
|
2022-11-15 22:48:02 +08:00 |
|
Joe Zhou
|
075b98291b
|
Return spider in setEmptySleepTime itself for chainning.
|
2022-11-12 11:06:08 +08:00 |
|
Joe Zhou
|
ac912e8f1f
|
Revise QueueScheduler to support capacity-restricted.
|
2022-11-12 10:17:36 +08:00 |
|
Sutra Zhou
|
d2b2eed9df
|
Pass the task to onSuccess & onError.
|
2022-10-19 22:15:41 +08:00 |
|
Sutra Zhou
|
5f80e02abd
|
Interrupt current thread.
|
2022-10-19 22:08:38 +08:00 |
|
Sutra Zhou
|
126c32ecd0
|
Fix compatible issue.
|
2022-10-16 07:21:35 +08:00 |
vio.ao
|
e7a7fbeeeb
|
Enhance Jsoup could parse tr td tag directly
|
2022-10-01 18:23:15 +08:00 |
|
vio.ao
|
7a62a6cb45
|
Revert "Revert "Common the downloader status process and pass error information when …""
This reverts commit acfbd7b883.
|
2022-10-01 17:33:11 +08:00 |
|
Sutra Zhou
|
acfbd7b883
|
Revert "Common the downloader status process and pass error information when …"
|
2022-10-01 10:37:09 +08:00 |
|
vio.ao
|
d01f26333b
|
Common the downloader status process and pass error information when onError
|
2022-10-01 00:21:17 +08:00 |
David Hsing
|
54da7af17e
|
change dependency versions into properties
change dependency versions into properties
update commons-collections from 3.x to 4.4
|
2022-05-03 17:42:42 +08:00 |
|
Sutra Zhou
|
692605bd75
|
Merge branch 'develop' of github.com:code4craft/webmagic into develop
|
2021-10-24 23:20:56 +08:00 |
|
Sutra Zhou
|
34da2fb3a0
|
Make PageProcessor#getSite be default method. Closes #1040.
|
2021-10-24 23:20:38 +08:00 |
carl.don:tjr
|
fcdb9074d6
|
格式化 Spider.run 代码
|
2021-08-04 18:23:04 +08:00 |
|
carl.don:tjr
|
ab5d81a6b6
|
perfect Spider.run to avoid some rare concurrent issue, change the Spider.emptySleepTime to long type
|
2021-08-04 17:17:22 +08:00 |
|
Sutra Zhou
|
be6f5ff771
|
Add missing @Deprecated annotations.
|
2021-03-22 18:21:59 +08:00 |
|
Sutra Zhou
|
4e8a086dae
|
Pass exception to onError. Fixes #1005.
|
2021-03-22 18:21:10 +08:00 |
|
Sutra Zhou
|
0e01550a79
|
Upgrade dependencies, including the jedis from 2.9.3 to 3.4.1.
|
2021-01-06 03:21:10 +08:00 |
|
Sutra Zhou
|
ab6ff7f809
|
Revert "pageCount修改"
This reverts commit 9a71f0ac92.
|
2021-01-02 20:33:32 +08:00 |
|
Sutra Zhou
|
30daec4803
|
Revert "提供出现某种异常刷新代理，异常可配置"
This reverts commit 4a6441e7c5.
|
2021-01-02 20:33:17 +08:00 |
|
Sutra Zhou
|
d0843bee0d
|
Revert "简化代码"
This reverts commit 9cc5287743.
|
2021-01-02 20:32:35 +08:00 |
|
Sutra Zhou
|
5ceccc62e0
|
Revert "提供异常刷新httpClient，异常可配置，重写getHttpClient代码"
This reverts commit 19465089c3.
|
2021-01-02 20:31:53 +08:00 |
|
Sutra Zhou
|
33e3fcdf22
|
Revert "代理接口的修改，提供刷星代理API。downloader 下载错误时，提供request,exception,proxyProvider三个参数，"
This reverts commit ba69eba669.
|
2021-01-02 20:27:28 +08:00 |
|
Sutra Zhou
|
c489647c4b
|
Revert " Downloader 提供刷新组件的api,方便在spider中操作"
This reverts commit 2e2a0fdf3e.
|
2021-01-02 20:15:10 +08:00 |
|
Sutra Zhou
|
4bedd97267
|
Revert " 刷新代理api重构，需要提供旧代理，如果依然是旧代理，才进行刷新，防止应延迟响应造成的过度刷新"
This reverts commit 0aa2c3949d.
|
2021-01-02 20:14:02 +08:00 |
|
Sutra Zhou
|
3f756c9325
|
Revert " 代理功能扩展，对原代理提供商进行拆分，加入lombok"
This reverts commit 33906e36f4.
|
2021-01-02 20:14:01 +08:00 |
|
Sutra Zhou
|
aabc5584b8
|
Revert " bug修改，对结果提供缓存能力"
This reverts commit f68795d7dd.
|
2021-01-02 20:13:53 +08:00 |
|
Sutra Zhou
|
328c3e0d7d
|
Remove useless imports to fix build.
|
2021-01-02 19:41:05 +08:00 |
|
Sutra Zhou
|
1d536cf705
|
Merge pull request #976 from yaoqiangpersonal/master
主要是对代理的功能进行了增加和修改
|
2020-12-29 17:27:42 +08:00 |
yao
|
f68795d7dd
|
bug修改，对结果提供缓存能力
|
2020-12-29 16:54:38 +08:00 |
|
yao
|
33906e36f4
|
代理功能扩展，对原代理提供商进行拆分，加入lombok
|
2020-12-29 16:18:43 +08:00 |
|
yao
|
0aa2c3949d
|
刷新代理api重构，需要提供旧代理，如果依然是旧代理，才进行刷新，防止应延迟响应造成的过度刷新
|
2020-12-22 18:19:37 +08:00 |
|
yao
|
2e2a0fdf3e
|
Downloader 提供刷新组件的api,方便在spider中操作
|
2020-12-21 18:08:55 +08:00 |
|
yao
|
19465089c3
|
提供异常刷新httpClient，异常可配置，重写getHttpClient代码
|
2020-12-21 16:02:35 +08:00 |
|
yao
|
9cc5287743
|
简化代码
|
2020-12-21 14:58:01 +08:00 |
|
yao
|
4a6441e7c5
|
提供出现某种异常刷新代理，异常可配置
|
2020-12-21 14:52:25 +08:00 |