2622b448b8
fix test
2017-04-08 20:09:43 +08:00
b06a248c00
fix test
2017-04-08 20:06:04 +08:00
1cfbd13aae
refacor in httpclientdownloader
2017-04-08 20:04:56 +08:00
83ada9749e
fix test
2017-04-08 12:16:34 +08:00
fe95a6842f
Request再次重构:去掉params，仅保留HttpRequestBody
2017-04-08 12:12:39 +08:00
395396c68e
增加HttpRequestBody
2017-04-08 11:59:52 +08:00
c93a8a2722
修复字符编码检测BUG
2017-03-31 18:27:18 +08:00
74110e6ec5
remove useless file
2017-03-30 09:19:00 +08:00
b100dfe273
update version
2017-03-30 09:18:12 +08:00
1c24baa8d1
Request支持设置header与cookie
...
新增POST请求时,XML、JSON参数支持
Page支持获取响应header
2017-03-29 02:27:06 +08:00
6bd197859b
fix test
2017-03-27 09:55:50 +08:00
f23e138c72
add response headers to Page #508
2017-03-27 09:52:25 +08:00
c13110c4cb
fix samples
2017-03-21 07:53:43 +08:00
c51ac6017c
remove Site.addStartRequest() etc. #494
2017-03-21 07:32:34 +08:00
68050fc88e
test pass
2017-03-21 07:19:12 +08:00
474b7c9d57
refactor
2017-03-20 23:13:49 +08:00
25c81013ca
new proxy pool api
2017-03-20 22:47:24 +08:00
46297deaa1
HttpUriRequestConverter
2017-03-20 22:43:32 +08:00
1d86f7c048
compile passed in httpclientDownloader
2017-03-20 22:40:14 +08:00
b71f379512
fix
2017-03-18 12:18:00 +08:00
a7f9e7cad5
重构一部分httpclient
2017-03-18 12:16:21 +08:00
221c155060
move release connection before return proxy #396
2017-03-18 11:15:36 +08:00
68beff42c5
add test #493
2017-03-18 11:01:30 +08:00
79522f941e
Bug, add null check to site in HttpClientDownloader & HttpClientGenerator
2017-03-17 14:10:54 +08:00
e9341d0291
complete test #447
2017-03-17 07:54:28 +08:00
e7d35c4846
add params to all method of request #447
2017-03-17 07:18:05 +08:00
75bad591d7
rewrite hashCode and equals for params #447
2017-03-17 07:10:14 +08:00
11c32669b2
Merge pull request #447 from xbynet/master
...
简化POST参数设置.
2017-03-17 07:06:46 +08:00
aa01e27779
change constructor for Proxy to public #490
2017-03-17 07:02:02 +08:00
791520e6a0
fix a bug of RegexSelector when regex has zero-width assertions.
2017-03-17 00:06:15 +08:00
c175ea88c0
#more test #484
2017-03-11 11:43:18 +08:00
9b964c0a99
test for #484
2017-03-11 11:41:01 +08:00
fc702fd3b6
introduce mockito for test
2017-03-11 11:31:15 +08:00
5215a492cc
remove duplicate check for POST request #484
2017-03-11 11:26:13 +08:00
0a1fb19052
add tests #483
2017-03-11 10:56:31 +08:00
a2e7f0004b
Merge branch 'master' of github.com:code4craft/webmagic
2017-03-11 10:52:54 +08:00
ef32571821
rewrite Request.equals and hashCode, add Method to check #483
2017-03-11 10:52:39 +08:00
8b8f535c30
refactor:extract charset detect to utils
2017-03-11 10:43:10 +08:00
e645524ad2
fix bug,set ExecutorService
2017-03-04 20:57:29 +08:00
a872a6480e
fix code sample for github #348
2017-02-25 22:46:29 +08:00
1d2171805f
add test for #228
2017-02-25 22:30:48 +08:00
bbe0b52ddd
remove synchronized in QueueScheduler #410
2017-02-25 19:55:45 +08:00
ad69963005
remove synchronize in Page #411
2017-02-25 19:42:12 +08:00
3a796b9413
remove duplicate code #421
2017-02-25 12:01:12 +08:00
42f1018010
remove messy code
2017-02-21 14:08:05 +08:00
650468c0e4
解决POST中文参数乱码问题
2017-01-22 18:04:22 +08:00
aaccc93215
new version
2017-01-21 12:04:12 +08:00
3e633c6871
version
2017-01-21 11:51:14 +08:00
f45e2f118b
for release
2017-01-21 11:38:36 +08:00
d60615f503
修复使用startUrls没有设置domain导致使用cookie空指针的问题#438
2017-01-21 11:29:42 +08:00
407fbb6130
refactor logger#445
2017-01-21 11:05:54 +08:00
0dc26c8ca0
optimize code.
2017-01-20 14:03:26 +08:00
4f76d62d4f
Merge pull request #444 from ckex/develop
...
绕过安全证书
2017-01-18 23:42:51 +08:00
e4af05a6f2
绕过安全证书
2017-01-18 17:28:01 +08:00
c23627bf63
解决post/redirect/post 302跳转问题
2017-01-17 00:07:01 +08:00
d69204b919
0.6.0
2016-12-18 11:45:43 +08:00
9bdb48b2d0
version 0.6.0
2016-12-18 11:20:28 +08:00
eeb607fd0e
将Spider.processRequest()抛出异常改回原来的逻辑
2016-12-18 11:04:58 +08:00
97592d6720
Version 0.6.0
2016-12-18 10:58:24 +08:00
00dfebbceb
#424 remove guava dep and add fix docs
2016-12-18 10:45:50 +08:00
c2531c6817
clean dependency
2016-12-18 08:34:46 +08:00
a960a39c44
fix compile error for example change
2016-12-18 08:32:14 +08:00
7476ceccee
more stable test
2016-12-18 08:15:26 +08:00
5ce3fdfe5a
some refactor in log
2016-12-18 08:15:09 +08:00
98163a3e40
update examples
2016-12-18 07:46:18 +08:00
b090dcd20d
sepcific error page for HttpClientDownloaderTest to avoid test error when local port is available
2016-12-18 07:15:06 +08:00
8f942d6fe2
#419 修复抓取https链接线程无法结束导致进程一直运行的问题
2016-12-18 06:56:01 +08:00
dafd2b77ff
fix GithubRepoPageProcessor in example
2016-11-24 08:18:06 +08:00
cfed860fb9
Merge branch 'master' of github.com:code4craft/webmagic
2016-11-22 17:00:27 +08:00
2189aab652
fix test
2016-11-22 16:58:49 +08:00
1491033534
Merge pull request #377 from jerry-sc/monitor-bug
...
fix the monitor bug which the spider will terminate when a seed url with port
2016-11-19 13:01:30 +08:00
507556d0aa
fix test: ProxyTest.testProxy() do not load exist proxy config
2016-11-19 12:54:39 +08:00
e56b8c3efc
fix the monitor bug which the spider will terminate when a seed url with port
2016-09-22 22:36:18 +08:00
448e528140
update StringUtils to apache lang3 #314
2016-05-24 13:33:17 +08:00
3e33959b7a
#319 fix javadoc
2016-05-24 13:17:35 +08:00
8730e3e97a
Merge branch 'fix' of git://github.com/kapsterio/webmagic into kapsterio-fix
2016-05-08 20:46:22 +08:00
2400ff7e1a
resovle conflict
2016-05-08 20:31:43 +08:00
b7f3c4bba0
Merge branch 'master' of git://github.com/hepan/webmagic into hepan-master
2016-05-08 20:27:47 +08:00
d8f978fd20
fix test in JsonPathSelectorTest #289
2016-05-08 19:32:03 +08:00
61c28a0130
refactor on proxypool
2016-05-08 17:53:15 +08:00
b871b210c5
Merge branch 'proxy-strategy' of github.com:EdwardsBean/webmagic into EdwardsBean-proxy-strategy
2016-05-08 17:53:02 +08:00
b5413368de
update ut
2016-05-08 16:23:41 +08:00
83c27ebbc4
增加IP代理认证功能
2016-05-08 16:17:58 +08:00
ca072c5575
fix URL regex in GithubRepoPageProcessor #305
2016-05-08 12:09:45 +08:00
89c6e52863
代理增加用户名密码认证
2016-04-13 15:16:57 +08:00
047cb8ff8f
updated versions to 0.5.4-SNAPSHOT
2016-04-01 14:51:59 +08:00
6b179c3d55
这个改动的原因基于两点：1）代理归还给代理池的时机应该是执行完http请求后就要尽早归还 2)http代理应该是HttpClientDownloader该考虑的事，不应该有Spider来处理，Spider并不知道它的downloader是个HttpClientDownloader
2016-03-12 20:09:41 +08:00
5f106c9c69
当page为null时，意味着非正常的响应状态，应该抛出异常，否则SpiderListener的onSuccess方法和onError方法都会执行
2016-03-12 20:03:27 +08:00
c0b8e8f8ae
remove .classpath .project
2016-01-22 14:58:22 +08:00
a8e6de4b90
Merge branch 'master' of git.oschina.net:flashsword20/webmagic
2016-01-22 10:16:58 +08:00
0fd4623f0a
Merge branch 'osc'
2016-01-21 19:33:30 +08:00
ce5495ecd5
remove useless files
2016-01-21 19:31:50 +08:00
8265c7dade
remove submodules for relase
2016-01-21 19:25:13 +08:00
7edfa26f90
complete javadoc
2016-01-21 18:34:07 +08:00
8b90b91e33
complete some javadoc
2016-01-21 18:14:10 +08:00
2b556cf053
update verison to 0.5.3-SNAPSHOT
2016-01-21 18:05:56 +08:00
9c5716a543
complete javadoc
2016-01-21 18:05:12 +08:00
db3cbf6ca5
update version to 0.5.3-SNAPSHOT
2016-01-21 17:58:36 +08:00
81ce1ffc5f
fix ignore
2016-01-21 12:36:49 +08:00
93764fa2c9
ignore some test
2016-01-21 12:28:32 +08:00
5706bb90af
update xsoup to 0.3.1
2016-01-20 12:59:11 +08:00
7586e3d75c
add some test for github repo downloader
2016-01-19 08:05:53 +08:00
90e14b31b0
修正FileCacheQueueScheduler导致程序不能正常结束和未关闭流
...
FileCacheQueueScheduler中开启了一个线程周期运行来保存数据但在爬虫结束后没有关闭导致程序无法结束，以及没有关闭io流。
解决方法：
让FileCacheQueueScheduler实现Closable接口，在close方法中关闭线程以及流。
在Spider的close方法中添加对scheduler的关闭操作。
2015-11-12 23:10:20 +08:00
56e0cd513a
compile error fix
2015-04-15 23:21:06 +08:00
c5740b1840
change assert #200
2015-04-15 08:32:08 +08:00
67eb632f4d
test for issue #200
2015-04-15 08:31:45 +08:00
590561a6e4
修正site.setHttpProxy()不起作用的bug
2015-03-09 15:54:15 +08:00
19474e4716
add SimpleProxyPool and IProxyPool
2015-02-28 17:50:10 +08:00
4978665633
add retry sleep time
2015-01-21 13:30:02 +08:00
8ffc1a7093
add NPE check for POST method
2015-01-13 14:10:00 +08:00
bc666e927d
Update Site.java
...
setCycleRetryTimes的javadoc是这么说的:Set cycleRetryTimes times when download fail, 0 by default. Only work in RedisScheduler.
而通过查看源码发现似乎并没有做限制,即只能用于RedisScheduler. 故想问一下该javadoc是否过时了?
2014-09-12 12:42:57 +08:00
147401ce5e
remove duplicate setPath in ProxyPool
2014-09-09 22:58:44 +08:00
e7668e01b8
fix SourceRegion error and add some tests on it #144
2014-08-21 14:29:06 +08:00
4446669c24
fix test
2014-08-18 10:54:24 +08:00
9866297ec4
Disable jsoup entity escape by Default. Set Html.DISABLE_HTML_ENTITY_ESCAPE to false to enable it. #149
2014-08-14 08:04:56 +08:00
4e6e946dd7
more friendly exception message in PlainText #144
2014-08-13 10:02:16 +08:00
af9939622b
move thread package out of selector (because it is add by mistake at the beginning)
2014-06-25 18:19:50 +08:00
eae37c868b
new sample
2014-06-10 17:38:54 +08:00
b3a282e58d
some fix for tests #130
2014-06-10 00:05:30 +08:00
074d767f45
Merge branch 'proxy' of github.com:yxssfxwzy/webmagic into yxssfxwzy-proxy
2014-06-09 23:51:36 +08:00
2f89cfc31a
add test and fix bug of proxy module
2014-06-09 13:32:02 +08:00
eb89d66566
fix test
2014-06-04 22:28:27 +08:00
5e8ca02ec6
contributor
2014-06-04 22:26:56 +08:00
8c33be48a6
Merge branch 'stable' of github.com:code4craft/webmagic
2014-06-04 17:37:45 +08:00
5f8c3fd5c5
update version
2014-06-04 17:33:30 +08:00
7a64847a3c
Bugfix: selector does not works well in element #113
2014-06-03 20:03:33 +08:00
8d67fd0357
change back return proxy from spider to httpclientdownloader #128
2014-05-28 08:08:51 +08:00
40bf8ca58f
change return proxy from spider to httpclientdownloader #128
2014-05-28 07:57:42 +08:00
1f21d9cc14
spell mistake fix #128
2014-05-28 07:29:19 +08:00
e310139d00
Merge pull request #128 from yxssfxwzy/proxy
...
多个代理的管理
2014-05-28 07:22:08 +08:00
b165090434
Bugfix:Type convert error in JsonPathSelector #129
2014-05-27 21:19:22 +08:00
a5d1b56e44
fix ut #113
2014-05-27 18:07:53 +08:00
3939074a23
Bugfix: nodes() only return the first element #113
2014-05-27 17:53:06 +08:00
41c2ea9498
refactor of selectable cont' #113
...
1. remove lazy init of Html
2. rename strings to sourceTexts for better meaning
3. make getSourceTexts abstract and DO NOT always store strings
4. instead store parsed elements of document in HtmlNode
2014-05-27 17:34:19 +08:00
f9825c214a
refactor selectable for html fragment #113
2014-05-27 16:00:51 +08:00
03d26c169b
Enhance auto charset detect #126
...
1. Only read from content once to fix stream closed exception
2. invite moco as server test
2014-05-26 17:45:30 +08:00
c146e2c7b4
add proxy pool
2014-05-19 15:59:31 +08:00
21982d3460
remove cpdetector temporary #126
2014-05-14 23:52:27 +08:00
fcbfb75608
修改自动从网页中获取字符的代码块，抽取出来成为单独的方法。
2014-05-14 19:14:42 +08:00
95494d3c4d
增加处理meta的逻辑。
...
遗留：
3、网页没有指定编码的情况下，需要采用cpdetector，但目前cpdetector这个在Maven的中央库里面没有，不清楚如何解决。
2014-05-14 14:53:54 +08:00
dde2d89bbe
Ignore content in json when bracket when remove padding #124
2014-05-08 23:37:18 +08:00
259f0a16c5
Update FilePipeline.java
2014-05-06 18:33:00 +08:00
26d38851b5
add charset to Writer
2014-05-06 18:28:50 +08:00
7668731f08
update version to snapshot
2014-05-05 07:03:55 +08:00
182dd51689
Merge branch 'stable' of github.com:code4craft/webmagic
2014-05-03 06:19:11 +08:00
81e6e772ac
versions back to 0.5.1
2014-05-03 06:18:57 +08:00
feb604da87
Merge branch 'stable' of github.com:code4craft/webmagic
2014-05-03 06:14:54 +08:00
358e906379
[maven-release-plugin] prepare for next development iteration
2014-05-03 00:00:13 +08:00
470750fc0d
[maven-release-plugin] prepare release WebMagic-0.5.1
2014-05-02 23:59:55 +08:00
01aec7e1ab
extension point of geturl #118
2014-05-02 23:23:23 +08:00
ec1c2e8cbc
test and so on
2014-05-02 23:19:11 +08:00
4f22f1210e
some bug fix #118
2014-05-02 20:38:49 +08:00
56f033ce8d
set setDuplicateRemover for chain api #118
2014-05-02 20:21:23 +08:00
d1140b9e29
add bloom filter for scheduler #118
2014-05-02 20:20:22 +08:00
8e4814bdc5
fix path seperator
2014-05-02 17:06:34 +08:00
e8d4a9be2b
fix remove duplicate error #117
2014-04-29 20:32:06 +08:00
04ade75606
Merge branch 'stable' of github.com:code4craft/webmagic
...
Conflicts:
README.md
pom.xml
webmagic-avalon/pom.xml
webmagic-core/pom.xml
webmagic-extension/pom.xml
webmagic-lucene/pom.xml
webmagic-samples/pom.xml
webmagic-saxon/pom.xml
webmagic-scripts/pom.xml
webmagic-selenium/pom.xml
2014-04-27 15:03:15 +08:00
a08d8cb167
update verion
2014-04-27 14:59:48 +08:00
42a2676e8c
update version
2014-04-27 14:56:21 +08:00
c25b32f1ca
[maven-release-plugin] prepare for next development iteration
2014-04-27 12:52:27 +08:00
7ff83bb11a
[maven-release-plugin] prepare release WebMagic-0.5.0
2014-04-27 12:52:12 +08:00
1104122979
more abstraction in scheduler
2014-04-27 09:30:01 +08:00
2770811a10
update monitor example
2014-04-26 11:24:22 +08:00
5ecd909ef2
add timeout for wait/notify #111
2014-04-25 19:41:36 +08:00
c7afdb516e
remove thread utils #110
2014-04-25 18:44:45 +08:00
17e95f2a7f
comments
2014-04-25 18:39:01 +08:00
05eb7831b6
refactor and comments #110
2014-04-25 18:27:40 +08:00
375e64e845
more monitor status
2014-04-25 18:10:14 +08:00
018061d2cd
fix error in thread pool
2014-04-25 18:01:02 +08:00
cdc423f2bf
log
2014-04-25 17:41:41 +08:00
c6661899fd
new thread pool #110
2014-04-25 17:33:48 +08:00
179baa7a22
return when page is null
2014-04-25 16:07:41 +08:00
0336f4cdb4
remove IllegalStateException when download error for less error log
2014-04-25 16:06:29 +08:00
11ba5beb42
[refactor]move monitor to webmagic-extension #98
2014-04-25 13:17:13 +08:00
d61f65cef8
update mbean to mxbean #98
2014-04-25 11:31:43 +08:00
ad6a273b12
update test url
2014-04-25 11:28:35 +08:00
30af23d003
split monitor to server and client mode #98
2014-04-25 11:25:52 +08:00
ced79630d3
specify jndi and jmx #98
2014-04-25 11:11:15 +08:00
95d3802e77
add formdata support for post request #108
2014-04-24 11:48:58 +08:00
f49bb877c8
clean some code #109
2014-04-24 11:38:13 +08:00
e1aaf1dd11
fix mistake of guava Table #109
2014-04-24 11:05:49 +08:00
8ba2da146c
request method #108 and more cookie #109 config
2014-04-24 10:51:37 +08:00
b06aa489fb
[BugFix]Only one url from sourceRegion can be extracted #107
2014-04-18 17:48:26 +08:00
08fa3b01c1
when download error, throw an exception instead of calling onError and returning peacefully. #105
2014-04-17 17:53:12 +08:00
27b37e8164
extension point and sample for JMX support #98
2014-04-17 08:12:37 +08:00
a5db6cf292
some monitor and JMX support #98
2014-04-17 00:35:09 +08:00
f39aa435cf
add null check #104
2014-04-16 19:46:32 +08:00
42bbe40a37
[Bugfix]Urls will be lost when call setScheduler() #104
2014-04-16 19:45:17 +08:00
163773af6b
combine two try-catch block into one, make it cleaner.
2014-04-16 16:05:08 +08:00
ec446277b1
some refactor in httpclientdownloader
2014-04-15 15:30:37 +08:00
a03f6a8431
eclipse project
2014-04-15 07:44:43 +08:00
4a035e729a
extension point for LocalDuplicatedRemovedScheduler #95
2014-04-13 23:31:13 +08:00
b249e49748
[Bugfix]loop error when add TargetRequest #99
2014-04-13 23:04:09 +08:00
da2f023c12
Merge pull request #96 from ouyanghuangzheng/master
...
修改了Spider 和site 几处注释
2014-04-13 13:12:12 +08:00
f7950ebcab
fix tests
2014-04-13 13:00:31 +08:00
32ba1b8889
修复几处注释问题
2014-04-13 12:41:15 +08:00
84b897f83b
update AngularJSProcessor
2014-04-13 12:20:57 +08:00
03c251237b
add Json parse support
2014-04-13 10:23:00 +08:00
644e8d1f72
同步官方源码
2014-04-12 22:32:22 +08:00
969ad1766b
change logger style to slf4j for cleaner code
2014-04-06 21:32:20 +08:00
yihua.huang
xbynet
wuyifan
mei
Ckex.zha
Jerry
Jon
hepan
Linker Lin
zhangheng09
x1ny
高军
edwardsbean
zhugw
zwf
fengwuze
ywooer
Bo LIANG
愤怒的番茄