Mobai
/
magic-Dependency
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
997 Commits
1 Branch
0 Tags
14 MiB
Commit Graph

491 Commits (3266ea15cab0218551bc7577ccc754f2feb30799)

Author SHA1 Message Date
yihua.huang b100dfe273 update version 2017-03-30 09:18:12 +08:00
xbynet@outlook.com 1c24baa8d1 Request支持设置header与cookie 
新增POST请求时,XML、JSON参数支持
Page支持获取响应header
 2017-03-29 02:27:06 +08:00
yihua.huang 6bd197859b fix test 2017-03-27 09:55:50 +08:00
yihua.huang f23e138c72 add response headers to Page #508 2017-03-27 09:52:25 +08:00
yihua.huang c13110c4cb fix samples 2017-03-21 07:53:43 +08:00
yihua.huang c51ac6017c remove Site.addStartRequest() etc. #494 2017-03-21 07:32:34 +08:00
yihua.huang 68050fc88e test pass 2017-03-21 07:19:12 +08:00
yihua.huang 474b7c9d57 refactor 2017-03-20 23:13:49 +08:00
yihua.huang 25c81013ca new proxy pool api 2017-03-20 22:47:24 +08:00
yihua.huang 46297deaa1 HttpUriRequestConverter 2017-03-20 22:43:32 +08:00
yihua.huang 1d86f7c048 compile passed in httpclientDownloader 2017-03-20 22:40:14 +08:00
yihua.huang b71f379512 fix 2017-03-18 12:18:00 +08:00
yihua.huang a7f9e7cad5 重构一部分httpclient 2017-03-18 12:16:21 +08:00
yihua.huang 221c155060 move release connection before return proxy #396 2017-03-18 11:15:36 +08:00
yihua.huang 68beff42c5 add test #493 2017-03-18 11:01:30 +08:00
wuyifan 79522f941e Bug, add null check to site in HttpClientDownloader & HttpClientGenerator 2017-03-17 14:10:54 +08:00
yihua.huang e9341d0291 complete test #447 2017-03-17 07:54:28 +08:00
yihua.huang e7d35c4846 add params to all method of request #447 2017-03-17 07:18:05 +08:00
yihua.huang 75bad591d7 rewrite hashCode and equals for params #447 2017-03-17 07:10:14 +08:00
Yihua Huang 11c32669b2 Merge pull request #447 from xbynet/master 
简化POST参数设置.
 2017-03-17 07:06:46 +08:00
yihua.huang aa01e27779 change constructor for Proxy to public #490 2017-03-17 07:02:02 +08:00
mei 791520e6a0 fix a bug of RegexSelector when regex has zero-width assertions. 2017-03-17 00:06:15 +08:00
yihua.huang c175ea88c0 #more test #484 2017-03-11 11:43:18 +08:00
yihua.huang 9b964c0a99 test for #484 2017-03-11 11:41:01 +08:00
yihua.huang fc702fd3b6 introduce mockito for test 2017-03-11 11:31:15 +08:00
yihua.huang 5215a492cc remove duplicate check for POST request #484 2017-03-11 11:26:13 +08:00
yihua.huang 0a1fb19052 add tests #483 2017-03-11 10:56:31 +08:00
yihua.huang a2e7f0004b Merge branch 'master' of github.com:code4craft/webmagic 2017-03-11 10:52:54 +08:00
yihua.huang ef32571821 rewrite Request.equals and hashCode, add Method to check #483 2017-03-11 10:52:39 +08:00
yihua.huang 8b8f535c30 refactor:extract charset detect to utils 2017-03-11 10:43:10 +08:00
Ckex.zha e645524ad2 fix bug,set ExecutorService 2017-03-04 20:57:29 +08:00
yihua.huang a872a6480e fix code sample for github #348 2017-02-25 22:46:29 +08:00
yihua.huang 1d2171805f add test for #228 2017-02-25 22:30:48 +08:00
yihua.huang bbe0b52ddd remove synchronized in QueueScheduler #410 2017-02-25 19:55:45 +08:00
yihua.huang ad69963005 remove synchronize in Page #411 2017-02-25 19:42:12 +08:00
yihua.huang 3a796b9413 remove duplicate code #421 2017-02-25 12:01:12 +08:00
yihua.huang 42f1018010 remove messy code 2017-02-21 14:08:05 +08:00
xbynet 650468c0e4 解决POST中文参数乱码问题 2017-01-22 18:04:22 +08:00
yihua.huang aaccc93215 new version 2017-01-21 12:04:12 +08:00
yihua.huang 3e633c6871 version 2017-01-21 11:51:14 +08:00
yihua.huang f45e2f118b for release 2017-01-21 11:38:36 +08:00
yihua.huang d60615f503 修复使用startUrls没有设置domain导致使用cookie空指针的问题#438 2017-01-21 11:29:42 +08:00
yihua.huang 407fbb6130 refactor logger#445 2017-01-21 11:05:54 +08:00
Ckex.zha 0dc26c8ca0 optimize code. 2017-01-20 14:03:26 +08:00
Yihua Huang 4f76d62d4f Merge pull request #444 from ckex/develop 
绕过安全证书
 2017-01-18 23:42:51 +08:00
Ckex.zha e4af05a6f2 绕过安全证书 2017-01-18 17:28:01 +08:00
xbynet@outlook.com c23627bf63 解决post/redirect/post 302跳转问题 2017-01-17 00:07:01 +08:00
yihua.huang d69204b919 0.6.0 2016-12-18 11:45:43 +08:00
yihua.huang 9bdb48b2d0 version 0.6.0 2016-12-18 11:20:28 +08:00
yihua.huang eeb607fd0e 将Spider.processRequest()抛出异常改回原来的逻辑 2016-12-18 11:04:58 +08:00
yihua.huang 97592d6720 Version 0.6.0 2016-12-18 10:58:24 +08:00
yihua.huang 00dfebbceb #424 remove guava dep and add fix docs 2016-12-18 10:45:50 +08:00
yihua.huang c2531c6817 clean dependency 2016-12-18 08:34:46 +08:00
yihua.huang a960a39c44 fix compile error for example change 2016-12-18 08:32:14 +08:00
yihua.huang 7476ceccee more stable test 2016-12-18 08:15:26 +08:00
yihua.huang 5ce3fdfe5a some refactor in log 2016-12-18 08:15:09 +08:00
yihua.huang 98163a3e40 update examples 2016-12-18 07:46:18 +08:00
yihua.huang b090dcd20d sepcific error page for HttpClientDownloaderTest to avoid test error when local port is available 2016-12-18 07:15:06 +08:00
yihua.huang 8f942d6fe2 #419 修复抓取https链接线程无法结束导致进程一直运行的问题 2016-12-18 06:56:01 +08:00
yihua.huang dafd2b77ff fix GithubRepoPageProcessor in example 2016-11-24 08:18:06 +08:00
yihua.huang cfed860fb9 Merge branch 'master' of github.com:code4craft/webmagic 2016-11-22 17:00:27 +08:00
yihua.huang 2189aab652 fix test 2016-11-22 16:58:49 +08:00
Yihua Huang 1491033534 Merge pull request #377 from jerry-sc/monitor-bug 
fix the monitor bug which the spider will terminate when a seed url with port
 2016-11-19 13:01:30 +08:00
yihua.huang 507556d0aa fix test: ProxyTest.testProxy() do not load exist proxy config 2016-11-19 12:54:39 +08:00
Jerry e56b8c3efc fix the monitor bug which the spider will terminate when a seed url with port 2016-09-22 22:36:18 +08:00
yihua.huang 448e528140 update StringUtils to apache lang3 #314 2016-05-24 13:33:17 +08:00
yihua.huang 3e33959b7a #319 fix javadoc 2016-05-24 13:17:35 +08:00
yihua.huang 8730e3e97a Merge branch 'fix' of git://github.com/kapsterio/webmagic into kapsterio-fix 2016-05-08 20:46:22 +08:00
yihua.huang 2400ff7e1a resovle conflict 2016-05-08 20:31:43 +08:00
yihua.huang b7f3c4bba0 Merge branch 'master' of git://github.com/hepan/webmagic into hepan-master 2016-05-08 20:27:47 +08:00
yihua.huang d8f978fd20 fix test in JsonPathSelectorTest #289 2016-05-08 19:32:03 +08:00
yihua.huang 61c28a0130 refactor on proxypool 2016-05-08 17:53:15 +08:00
yihua.huang b871b210c5 Merge branch 'proxy-strategy' of github.com:EdwardsBean/webmagic into EdwardsBean-proxy-strategy 2016-05-08 17:53:02 +08:00
yihua.huang b5413368de update ut 2016-05-08 16:23:41 +08:00
Jon 83c27ebbc4 增加IP代理认证功能 2016-05-08 16:17:58 +08:00
yihua.huang ca072c5575 fix URL regex in GithubRepoPageProcessor #305 2016-05-08 12:09:45 +08:00
hepan 89c6e52863 代理增加用户名密码认证 2016-04-13 15:16:57 +08:00
Linker Lin 047cb8ff8f updated versions to 0.5.4-SNAPSHOT 2016-04-01 14:51:59 +08:00
zhangheng09 6b179c3d55 这个改动的原因基于两点：1）代理归还给代理池的时机应该是执行完http请求后就要尽早归还 2)http代理应该是HttpClientDownloader该考虑的事，不应该有Spider来处理，Spider并不知道它的downloader是个HttpClientDownloader 2016-03-12 20:09:41 +08:00
zhangheng09 5f106c9c69 当page为null时，意味着非正常的响应状态，应该抛出异常，否则SpiderListener的onSuccess方法和onError方法都会执行 2016-03-12 20:03:27 +08:00
yihua.huang c0b8e8f8ae remove .classpath .project 2016-01-22 14:58:22 +08:00
yihua.huang a8e6de4b90 Merge branch 'master' of git.oschina.net:flashsword20/webmagic 2016-01-22 10:16:58 +08:00
yihua.huang 0fd4623f0a Merge branch 'osc' 2016-01-21 19:33:30 +08:00
yihua.huang ce5495ecd5 remove useless files 2016-01-21 19:31:50 +08:00
yihua.huang 8265c7dade remove submodules for relase 2016-01-21 19:25:13 +08:00
yihua.huang 7edfa26f90 complete javadoc 2016-01-21 18:34:07 +08:00
yihua.huang 8b90b91e33 complete some javadoc 2016-01-21 18:14:10 +08:00
yihua.huang 2b556cf053 update verison to 0.5.3-SNAPSHOT 2016-01-21 18:05:56 +08:00
yihua.huang 9c5716a543 complete javadoc 2016-01-21 18:05:12 +08:00
yihua.huang db3cbf6ca5 update version to 0.5.3-SNAPSHOT 2016-01-21 17:58:36 +08:00
yihua.huang 81ce1ffc5f fix ignore 2016-01-21 12:36:49 +08:00
yihua.huang 93764fa2c9 ignore some test 2016-01-21 12:28:32 +08:00
yihua.huang 5706bb90af update xsoup to 0.3.1 2016-01-20 12:59:11 +08:00
yihua.huang 7586e3d75c add some test for github repo downloader 2016-01-19 08:05:53 +08:00
x1ny 90e14b31b0 修正FileCacheQueueScheduler导致程序不能正常结束和未关闭流 
FileCacheQueueScheduler中开启了一个线程周期运行来保存数据但在爬虫结束后没有关闭导致程序无法结束，以及没有关闭io流。

解决方法：
让FileCacheQueueScheduler实现Closable接口，在close方法中关闭线程以及流。
在Spider的close方法中添加对scheduler的关闭操作。
 2015-11-12 23:10:20 +08:00
yihua.huang 56e0cd513a compile error fix 2015-04-15 23:21:06 +08:00
yihua.huang c5740b1840 change assert #200 2015-04-15 08:32:08 +08:00
yihua.huang 67eb632f4d test for issue #200 2015-04-15 08:31:45 +08:00
高军 590561a6e4 修正site.setHttpProxy()不起作用的bug 2015-03-09 15:54:15 +08:00
edwardsbean 19474e4716 add SimpleProxyPool and IProxyPool 2015-02-28 17:50:10 +08:00