webMagic-Mobai
/
magic-Dependency
mirror of https://gitea.qinmian.online/Mobai/magic-Dependency.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
1,190 Commits
1 Branch
0 Tags
14 MiB
Commit Graph

1190 Commits (5d5f3bf20e08e37b44b2807021b056cbc696a5f9)

Author SHA1 Message Date
yihua.huang b7f3c4bba0 Merge branch 'master' of git://github.com/hepan/webmagic into hepan-master 2016-05-08 20:27:47 +08:00
yihua.huang d8f978fd20 fix test in JsonPathSelectorTest #289 2016-05-08 19:32:03 +08:00
yihua.huang 25821c3c06 Merge branch 'EdwardsBean-proxy-strategy' 2016-05-08 17:53:25 +08:00
yihua.huang 61c28a0130 refactor on proxypool 2016-05-08 17:53:15 +08:00
yihua.huang b871b210c5 Merge branch 'proxy-strategy' of github.com:EdwardsBean/webmagic into EdwardsBean-proxy-strategy 2016-05-08 17:53:02 +08:00
yihua.huang b5413368de update ut 2016-05-08 16:23:41 +08:00
yihua.huang a8c9005381 Merge branch 'oxf1-master' 2016-05-08 16:18:29 +08:00
Jon 83c27ebbc4 增加IP代理认证功能 2016-05-08 16:17:58 +08:00
yihua.huang ca072c5575 fix URL regex in GithubRepoPageProcessor #305 2016-05-08 12:09:45 +08:00
hepan 89c6e52863 代理增加用户名密码认证 2016-04-13 15:16:57 +08:00
Linker Lin 047cb8ff8f updated versions to 0.5.4-SNAPSHOT 2016-04-01 14:51:59 +08:00
Linker Lin b9544424f2 Update pom.xml 
回滚。。。
我想，主要的问题还是主版本号没了 SNAPSHOT
 2016-04-01 14:36:15 +08:00
Linker Lin b05d605c4b Update pom.xml 
在纯净的环境中，需要做上面的改动。
 2016-04-01 14:19:28 +08:00
zhangheng09 6b179c3d55 这个改动的原因基于两点：1）代理归还给代理池的时机应该是执行完http请求后就要尽早归还 2)http代理应该是HttpClientDownloader该考虑的事，不应该有Spider来处理，Spider并不知道它的downloader是个HttpClientDownloader 2016-03-12 20:09:41 +08:00
zhangheng09 5f106c9c69 当page为null时，意味着非正常的响应状态，应该抛出异常，否则SpiderListener的onSuccess方法和onError方法都会执行 2016-03-12 20:03:27 +08:00
yihua.huang c0b8e8f8ae remove .classpath .project 2016-01-22 14:58:22 +08:00
yihua.huang ce89ebb2b4 remove en_doc zh_docs dir 2016-01-22 10:49:36 +08:00
yihua.huang b9ac76aa8b update readme 2016-01-22 10:48:44 +08:00
yihua.huang a0c74a6a26 readme in zh 2016-01-22 10:47:10 +08:00
yihua.huang cb61820d31 update spiderman link 2016-01-22 10:24:30 +08:00
yihua.huang a8e6de4b90 Merge branch 'master' of git.oschina.net:flashsword20/webmagic 2016-01-22 10:16:58 +08:00
yihua.huang cd08b230b8 update version to 0.5.3 2016-01-22 09:44:54 +08:00
yihua.huang 0fd4623f0a Merge branch 'osc' 2016-01-21 19:33:30 +08:00
yihua.huang ce5495ecd5 remove useless files 2016-01-21 19:31:50 +08:00
yihua.huang 8265c7dade remove submodules for relase 2016-01-21 19:25:13 +08:00
yihua.huang 972ccd3d56 update version to 0.5.3-SNAPSHOT 2016-01-21 19:03:07 +08:00
yihua.huang 7edfa26f90 complete javadoc 2016-01-21 18:34:07 +08:00
yihua.huang 8b90b91e33 complete some javadoc 2016-01-21 18:14:10 +08:00
yihua.huang 2b556cf053 update verison to 0.5.3-SNAPSHOT 2016-01-21 18:05:56 +08:00
yihua.huang 9c5716a543 complete javadoc 2016-01-21 18:05:12 +08:00
yihua.huang db3cbf6ca5 update version to 0.5.3-SNAPSHOT 2016-01-21 17:58:36 +08:00
yihua.huang 81ce1ffc5f fix ignore 2016-01-21 12:36:49 +08:00
yihua.huang 93764fa2c9 ignore some test 2016-01-21 12:28:32 +08:00
yihua.huang 5706bb90af update xsoup to 0.3.1 2016-01-20 12:59:11 +08:00
yihua.huang 7586e3d75c add some test for github repo downloader 2016-01-19 08:05:53 +08:00
yihua.huang 800f66c4cc Revert "remove some unkown config" 
This reverts commit 0e245c9896.
 2016-01-18 23:20:08 +08:00
yihua.huang 73ae7a1d52 remove ci for jdk6 2016-01-18 23:19:39 +08:00
yihua.huang 0e245c9896 remove some unkown config 2016-01-18 22:39:45 +08:00
yihua.huang 9ed06ccdf0 update surefire version 2016-01-18 22:31:15 +08:00
Yihua Huang 9d0eeb9000 Merge pull request #218 from bingoko/master 
添加PhantomJS无界面浏览器支持
 2016-01-16 19:38:01 +08:00
Yihua Huang 84b046e4c9 Merge pull request #227 from hsqlu/master 
update deprecated method
 2016-01-16 19:36:52 +08:00
Yihua Huang cfde3b7657 Merge pull request #237 from SpenceZhou/master 
Update RedisScheduler.java
 2015-12-02 22:17:00 +08:00
SpenceZhou 165e5a72eb Update RedisScheduler.java 
修改redisscheduler中获取爬取总数bug
 2015-12-02 17:10:42 +08:00
Yihua Huang 5f9e1a96f2 Merge pull request #233 from x1ny/master 
修正FileCacheQueueScheduler导致程序不能正常结束和未关闭流
 2015-11-17 15:06:08 +08:00
Yihua Huang 7d7eb033d3 Merge pull request #234 from chy996633/master 
知乎爬虫抓取
 2015-11-17 15:05:46 +08:00
chy996633 afd1617b58 知乎爬虫抓取 2015-11-13 11:56:46 +08:00
x1ny 90e14b31b0 修正FileCacheQueueScheduler导致程序不能正常结束和未关闭流 
FileCacheQueueScheduler中开启了一个线程周期运行来保存数据但在爬虫结束后没有关闭导致程序无法结束，以及没有关闭io流。

解决方法：
让FileCacheQueueScheduler实现Closable接口，在close方法中关闭线程以及流。
在Spider的close方法中添加对scheduler的关闭操作。
 2015-11-12 23:10:20 +08:00
Qiannan Lu 155215290f resolve issue #226 2015-09-18 09:30:01 +08:00
Qiannan Lu 21f81bb8c1 update deprecated method 2015-09-18 08:59:40 +08:00
bingoko 5d365f7bf4 update and validate pom.xml 
Update selenium and GhostDriver (PhantomJSDriver) to latest version.
 2015-07-11 15:43:49 +01:00