Mobai
/
magic-Dependency
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
1,023 Commits
1 Branch
0 Tags
14 MiB
Commit Graph

1023 Commits (ba1b4017a7399d7fa073b309d5376e5f3463214d)

Author SHA1 Message Date
yihua.huang 7476ceccee more stable test 2016-12-18 08:15:26 +08:00
yihua.huang 5ce3fdfe5a some refactor in log 2016-12-18 08:15:09 +08:00
yihua.huang 98163a3e40 update examples 2016-12-18 07:46:18 +08:00
yihua.huang 243ebc22fa #374 update httpclient version to 4.5.2 2016-12-18 07:15:40 +08:00
yihua.huang b090dcd20d sepcific error page for HttpClientDownloaderTest to avoid test error when local port is available 2016-12-18 07:15:06 +08:00
yihua.huang 8f942d6fe2 #419 修复抓取https链接线程无法结束导致进程一直运行的问题 2016-12-18 06:56:01 +08:00
Jsbd 1b886d48a2 新增PhantomJSDownloader构造函数，支持crawl.js路径自定义，因为当其他项目依赖此jar包时，runtime.exec()执行phantomjs命令时无使用法jar包中的crawl.js 2016-12-08 14:29:42 +08:00
Jsbd d1f2e65e5d 新增PhantomJSDownloader构造函数，支持crawl.js路径自定义，因为当其他项目依赖此jar包时，runtime.exec()执行phantomjs命令时无使用法jar包中的crawl.js 2016-12-08 14:28:48 +08:00
Jsbd f8a2328ead Merge pull request #1 from code4craft/master 
merge
 2016-12-02 17:22:55 +08:00
Yihua Huang 1987cd3ae1 Merge pull request #408 from code4craft/0.6.0 
groovy demo
 2016-12-02 14:20:46 +08:00
Yihua Huang 65fe2c4487 Merge pull request #407 from jsbd/master 
为PhantomJSDownloader添加新的构造函数，支持phantomjs自定义命令
 2016-12-02 14:05:58 +08:00
Jsbd ebc61363c8 为PhantomJSDownloader添加新的构造函数，支持phantomjs自定义命令 
为PhantomJSDownloader添加新的构造函数，支持phantomjs自定义命令
 example: 
   *    phantomjs.exe 支持windows环境
   *    phantomjs --ignore-ssl-errors=yes 忽略抓取地址是https时的一些错误
   *    /usr/local/bin/phantomjs 命令的绝对路径，避免因系统环境变量引起的IOException
 2016-12-02 10:17:46 +08:00
yihua.huang fdf39eb99d open in new page 2016-11-25 18:20:59 +08:00
yihua.huang 1e74494708 add related link 2016-11-25 18:19:32 +08:00
yihua.huang b92e6b04f0 #400 修复FileCacheQueueScheduler自己设置DuplicateRemover会导致NPE的问题 2016-11-25 08:30:24 +08:00
yihua.huang dafd2b77ff fix GithubRepoPageProcessor in example 2016-11-24 08:18:06 +08:00
yihua.huang cfed860fb9 Merge branch 'master' of github.com:code4craft/webmagic 2016-11-22 17:00:27 +08:00
yihua.huang 2189aab652 fix test 2016-11-22 16:58:49 +08:00
Yihua Huang 1491033534 Merge pull request #377 from jerry-sc/monitor-bug 
fix the monitor bug which the spider will terminate when a seed url with port
 2016-11-19 13:01:30 +08:00
Yihua Huang 228911b58c Merge pull request #370 from gyk001/master 
fixed #301 修复使用注解抽取JSON数据的问题
 2016-11-19 12:59:48 +08:00
yihua.huang 507556d0aa fix test: ProxyTest.testProxy() do not load exist proxy config 2016-11-19 12:54:39 +08:00
yihua.huang 55f131e5ef #380 update fastjson to 1.2.21 2016-11-19 12:53:59 +08:00
Jerry e56b8c3efc fix the monitor bug which the spider will terminate when a seed url with port 2016-09-22 22:36:18 +08:00
郭玉昆 700898fe8a fixed #301 修复使用注解抽取JSON数据的问题 2016-08-29 17:07:37 +08:00
Yihua Huang e22d6426fc Merge pull request #343 from Salon-sai/master 
add: redis scheduler with priority
 2016-08-18 13:28:25 +08:00
Salon.sai f89a6a6826 add: redis scheduler with priority 2016-07-05 16:29:01 +08:00
yihua.huang 448e528140 update StringUtils to apache lang3 #314 2016-05-24 13:33:17 +08:00
yihua.huang 3e33959b7a #319 fix javadoc 2016-05-24 13:17:35 +08:00
yihua.huang 3a6e246350 Merge branch 'kapsterio-fix' 2016-05-08 20:54:57 +08:00
yihua.huang 8730e3e97a Merge branch 'fix' of git://github.com/kapsterio/webmagic into kapsterio-fix 2016-05-08 20:46:22 +08:00
Yihua Huang 37cb43b667 Merge pull request #176 from lavenderx/master 
add PhantomJSDownloader
 2016-05-08 20:36:17 +08:00
yihua.huang 2400ff7e1a resovle conflict 2016-05-08 20:31:43 +08:00
yihua.huang 9de64ea0f2 Merge branch 'hepan-master' 2016-05-08 20:27:53 +08:00
yihua.huang b7f3c4bba0 Merge branch 'master' of git://github.com/hepan/webmagic into hepan-master 2016-05-08 20:27:47 +08:00
yihua.huang d8f978fd20 fix test in JsonPathSelectorTest #289 2016-05-08 19:32:03 +08:00
yihua.huang 25821c3c06 Merge branch 'EdwardsBean-proxy-strategy' 2016-05-08 17:53:25 +08:00
yihua.huang 61c28a0130 refactor on proxypool 2016-05-08 17:53:15 +08:00
yihua.huang b871b210c5 Merge branch 'proxy-strategy' of github.com:EdwardsBean/webmagic into EdwardsBean-proxy-strategy 2016-05-08 17:53:02 +08:00
yihua.huang b5413368de update ut 2016-05-08 16:23:41 +08:00
yihua.huang a8c9005381 Merge branch 'oxf1-master' 2016-05-08 16:18:29 +08:00
Jon 83c27ebbc4 增加IP代理认证功能 2016-05-08 16:17:58 +08:00
yihua.huang ca072c5575 fix URL regex in GithubRepoPageProcessor #305 2016-05-08 12:09:45 +08:00
hepan 89c6e52863 代理增加用户名密码认证 2016-04-13 15:16:57 +08:00
Linker Lin 047cb8ff8f updated versions to 0.5.4-SNAPSHOT 2016-04-01 14:51:59 +08:00
Linker Lin b9544424f2 Update pom.xml 
回滚。。。
我想，主要的问题还是主版本号没了 SNAPSHOT
 2016-04-01 14:36:15 +08:00
Linker Lin b05d605c4b Update pom.xml 
在纯净的环境中，需要做上面的改动。
 2016-04-01 14:19:28 +08:00
zhangheng09 6b179c3d55 这个改动的原因基于两点：1）代理归还给代理池的时机应该是执行完http请求后就要尽早归还 2)http代理应该是HttpClientDownloader该考虑的事，不应该有Spider来处理，Spider并不知道它的downloader是个HttpClientDownloader 2016-03-12 20:09:41 +08:00
zhangheng09 5f106c9c69 当page为null时，意味着非正常的响应状态，应该抛出异常，否则SpiderListener的onSuccess方法和onError方法都会执行 2016-03-12 20:03:27 +08:00
yihua.huang c0b8e8f8ae remove .classpath .project 2016-01-22 14:58:22 +08:00
yihua.huang ce89ebb2b4 remove en_doc zh_docs dir 2016-01-22 10:49:36 +08:00