ca072c5575
fix URL regex in GithubRepoPageProcessor #305
2016-05-08 12:09:45 +08:00
89c6e52863
代理增加用户名密码认证
2016-04-13 15:16:57 +08:00
047cb8ff8f
updated versions to 0.5.4-SNAPSHOT
2016-04-01 14:51:59 +08:00
b9544424f2
Update pom.xml
...
回滚。。。
我想，主要的问题还是主版本号没了 SNAPSHOT
2016-04-01 14:36:15 +08:00
b05d605c4b
Update pom.xml
...
在纯净的环境中，需要做上面的改动。
2016-04-01 14:19:28 +08:00
6b179c3d55
这个改动的原因基于两点：1）代理归还给代理池的时机应该是执行完http请求后就要尽早归还 2)http代理应该是HttpClientDownloader该考虑的事，不应该有Spider来处理，Spider并不知道它的downloader是个HttpClientDownloader
2016-03-12 20:09:41 +08:00
5f106c9c69
当page为null时，意味着非正常的响应状态，应该抛出异常，否则SpiderListener的onSuccess方法和onError方法都会执行
2016-03-12 20:03:27 +08:00
c0b8e8f8ae
remove .classpath .project
2016-01-22 14:58:22 +08:00
ce89ebb2b4
remove en_doc zh_docs dir
2016-01-22 10:49:36 +08:00
b9ac76aa8b
update readme
2016-01-22 10:48:44 +08:00
a0c74a6a26
readme in zh
2016-01-22 10:47:10 +08:00
cb61820d31
update spiderman link
2016-01-22 10:24:30 +08:00
a8e6de4b90
Merge branch 'master' of git.oschina.net:flashsword20/webmagic
2016-01-22 10:16:58 +08:00
cd08b230b8
update version to 0.5.3
2016-01-22 09:44:54 +08:00
0fd4623f0a
Merge branch 'osc'
2016-01-21 19:33:30 +08:00
ce5495ecd5
remove useless files
2016-01-21 19:31:50 +08:00
8265c7dade
remove submodules for relase
2016-01-21 19:25:13 +08:00
972ccd3d56
update version to 0.5.3-SNAPSHOT
2016-01-21 19:03:07 +08:00
7edfa26f90
complete javadoc
2016-01-21 18:34:07 +08:00
8b90b91e33
complete some javadoc
2016-01-21 18:14:10 +08:00
2b556cf053
update verison to 0.5.3-SNAPSHOT
2016-01-21 18:05:56 +08:00
9c5716a543
complete javadoc
2016-01-21 18:05:12 +08:00
db3cbf6ca5
update version to 0.5.3-SNAPSHOT
2016-01-21 17:58:36 +08:00
81ce1ffc5f
fix ignore
2016-01-21 12:36:49 +08:00
93764fa2c9
ignore some test
2016-01-21 12:28:32 +08:00
5706bb90af
update xsoup to 0.3.1
2016-01-20 12:59:11 +08:00
7586e3d75c
add some test for github repo downloader
2016-01-19 08:05:53 +08:00
800f66c4cc
Revert "remove some unkown config"
...
This reverts commit 0e245c9896
2016-01-18 23:20:08 +08:00
73ae7a1d52
remove ci for jdk6
2016-01-18 23:19:39 +08:00
0e245c9896
remove some unkown config
2016-01-18 22:39:45 +08:00
9ed06ccdf0
update surefire version
2016-01-18 22:31:15 +08:00
9d0eeb9000
Merge pull request #218 from bingoko/master
...
添加PhantomJS无界面浏览器支持
2016-01-16 19:38:01 +08:00
84b046e4c9
Merge pull request #227 from hsqlu/master
...
update deprecated method
2016-01-16 19:36:52 +08:00
cfde3b7657
Merge pull request #237 from SpenceZhou/master
...
Update RedisScheduler.java
2015-12-02 22:17:00 +08:00
165e5a72eb
Update RedisScheduler.java
...
修改redisscheduler中获取爬取总数bug
2015-12-02 17:10:42 +08:00
5f9e1a96f2
Merge pull request #233 from x1ny/master
...
修正FileCacheQueueScheduler导致程序不能正常结束和未关闭流
2015-11-17 15:06:08 +08:00
7d7eb033d3
Merge pull request #234 from chy996633/master
...
知乎爬虫抓取
2015-11-17 15:05:46 +08:00
afd1617b58
知乎爬虫抓取
2015-11-13 11:56:46 +08:00
90e14b31b0
修正FileCacheQueueScheduler导致程序不能正常结束和未关闭流
...
FileCacheQueueScheduler中开启了一个线程周期运行来保存数据但在爬虫结束后没有关闭导致程序无法结束，以及没有关闭io流。
解决方法：
让FileCacheQueueScheduler实现Closable接口，在close方法中关闭线程以及流。
在Spider的close方法中添加对scheduler的关闭操作。
2015-11-12 23:10:20 +08:00
155215290f
resolve issue #226
2015-09-18 09:30:01 +08:00
21f81bb8c1
update deprecated method
2015-09-18 08:59:40 +08:00
5d365f7bf4
update and validate pom.xml
...
Update selenium and GhostDriver (PhantomJSDriver) to latest version.
2015-07-11 15:43:49 +01:00
d3bbece202
Add PhantomJS support for selenium
...
The configuration file is config.ini
The dependencies are updated in pom.xml.
Update SeleniumDownloader and WebDriverPool to support PhantomJS.
NOTE: The versions of GhostDriver, Selenium, and PhantomJS are stable
and validated.
A GooglePlay Example is under samples package: GooglePlayProcessor.java
2015-07-11 15:34:21 +01:00
56e0cd513a
compile error fix
2015-04-15 23:21:06 +08:00
c5740b1840
change assert #200
2015-04-15 08:32:08 +08:00
67eb632f4d
test for issue #200
2015-04-15 08:31:45 +08:00
b30ca6ce1e
Merge pull request #198 from okuc/master
...
修正site.setHttpProxy()不起作用的bug
2015-03-09 16:58:52 +08:00
590561a6e4
修正site.setHttpProxy()不起作用的bug
2015-03-09 15:54:15 +08:00
19474e4716
add SimpleProxyPool and IProxyPool
2015-02-28 17:50:10 +08:00
05a1f39569
Merge pull request #193 from EdwardsBean/fix-mppipeline
...
Bug fix:MultiPagePipeline and DoubleKeyMap concurrent bug
2015-02-18 09:50:17 +08:00
yihua.huang
hepan
Linker Lin
Linker Lin
zhangheng09
SpenceZhou
chy996633
x1ny
Qiannan Lu
bingoko
高军
edwardsbean