2f89cfc31a
add test and fix bug of proxy module
2014-06-09 13:32:02 +08:00
eb89d66566
fix test
2014-06-04 22:28:27 +08:00
5e8ca02ec6
contributor
2014-06-04 22:26:56 +08:00
8c33be48a6
Merge branch 'stable' of github.com:code4craft/webmagic
2014-06-04 17:37:45 +08:00
5f8c3fd5c5
update version
2014-06-04 17:33:30 +08:00
7a64847a3c
Bugfix: selector does not works well in element #113
2014-06-03 20:03:33 +08:00
8d67fd0357
change back return proxy from spider to httpclientdownloader #128
2014-05-28 08:08:51 +08:00
40bf8ca58f
change return proxy from spider to httpclientdownloader #128
2014-05-28 07:57:42 +08:00
1f21d9cc14
spell mistake fix #128
2014-05-28 07:29:19 +08:00
e310139d00
Merge pull request #128 from yxssfxwzy/proxy
...
多个代理的管理
2014-05-28 07:22:08 +08:00
b165090434
Bugfix:Type convert error in JsonPathSelector #129
2014-05-27 21:19:22 +08:00
a5d1b56e44
fix ut #113
2014-05-27 18:07:53 +08:00
3939074a23
Bugfix: nodes() only return the first element #113
2014-05-27 17:53:06 +08:00
41c2ea9498
refactor of selectable cont' #113
...
1. remove lazy init of Html
2. rename strings to sourceTexts for better meaning
3. make getSourceTexts abstract and DO NOT always store strings
4. instead store parsed elements of document in HtmlNode
2014-05-27 17:34:19 +08:00
f9825c214a
refactor selectable for html fragment #113
2014-05-27 16:00:51 +08:00
03d26c169b
Enhance auto charset detect #126
...
1. Only read from content once to fix stream closed exception
2. invite moco as server test
2014-05-26 17:45:30 +08:00
c146e2c7b4
add proxy pool
2014-05-19 15:59:31 +08:00
21982d3460
remove cpdetector temporary #126
2014-05-14 23:52:27 +08:00
fcbfb75608
修改自动从网页中获取字符的代码块，抽取出来成为单独的方法。
2014-05-14 19:14:42 +08:00
95494d3c4d
增加处理meta的逻辑。
...
遗留：
3、网页没有指定编码的情况下，需要采用cpdetector，但目前cpdetector这个在Maven的中央库里面没有，不清楚如何解决。
2014-05-14 14:53:54 +08:00
dde2d89bbe
Ignore content in json when bracket when remove padding #124
2014-05-08 23:37:18 +08:00
259f0a16c5
Update FilePipeline.java
2014-05-06 18:33:00 +08:00
26d38851b5
add charset to Writer
2014-05-06 18:28:50 +08:00
7668731f08
update version to snapshot
2014-05-05 07:03:55 +08:00
182dd51689
Merge branch 'stable' of github.com:code4craft/webmagic
2014-05-03 06:19:11 +08:00
81e6e772ac
versions back to 0.5.1
2014-05-03 06:18:57 +08:00
feb604da87
Merge branch 'stable' of github.com:code4craft/webmagic
2014-05-03 06:14:54 +08:00
358e906379
[maven-release-plugin] prepare for next development iteration
2014-05-03 00:00:13 +08:00
470750fc0d
[maven-release-plugin] prepare release WebMagic-0.5.1
2014-05-02 23:59:55 +08:00
01aec7e1ab
extension point of geturl #118
2014-05-02 23:23:23 +08:00
ec1c2e8cbc
test and so on
2014-05-02 23:19:11 +08:00
4f22f1210e
some bug fix #118
2014-05-02 20:38:49 +08:00
56f033ce8d
set setDuplicateRemover for chain api #118
2014-05-02 20:21:23 +08:00
d1140b9e29
add bloom filter for scheduler #118
2014-05-02 20:20:22 +08:00
8e4814bdc5
fix path seperator
2014-05-02 17:06:34 +08:00
e8d4a9be2b
fix remove duplicate error #117
2014-04-29 20:32:06 +08:00
04ade75606
Merge branch 'stable' of github.com:code4craft/webmagic
...
Conflicts:
README.md
pom.xml
webmagic-avalon/pom.xml
webmagic-core/pom.xml
webmagic-extension/pom.xml
webmagic-lucene/pom.xml
webmagic-samples/pom.xml
webmagic-saxon/pom.xml
webmagic-scripts/pom.xml
webmagic-selenium/pom.xml
2014-04-27 15:03:15 +08:00
a08d8cb167
update verion
2014-04-27 14:59:48 +08:00
42a2676e8c
update version
2014-04-27 14:56:21 +08:00
c25b32f1ca
[maven-release-plugin] prepare for next development iteration
2014-04-27 12:52:27 +08:00
7ff83bb11a
[maven-release-plugin] prepare release WebMagic-0.5.0
2014-04-27 12:52:12 +08:00
1104122979
more abstraction in scheduler
2014-04-27 09:30:01 +08:00
2770811a10
update monitor example
2014-04-26 11:24:22 +08:00
5ecd909ef2
add timeout for wait/notify #111
2014-04-25 19:41:36 +08:00
c7afdb516e
remove thread utils #110
2014-04-25 18:44:45 +08:00
17e95f2a7f
comments
2014-04-25 18:39:01 +08:00
05eb7831b6
refactor and comments #110
2014-04-25 18:27:40 +08:00
375e64e845
more monitor status
2014-04-25 18:10:14 +08:00
018061d2cd
fix error in thread pool
2014-04-25 18:01:02 +08:00
cdc423f2bf
log
2014-04-25 17:41:41 +08:00
c6661899fd
new thread pool #110
2014-04-25 17:33:48 +08:00
179baa7a22
return when page is null
2014-04-25 16:07:41 +08:00
0336f4cdb4
remove IllegalStateException when download error for less error log
2014-04-25 16:06:29 +08:00
11ba5beb42
[refactor]move monitor to webmagic-extension #98
2014-04-25 13:17:13 +08:00
d61f65cef8
update mbean to mxbean #98
2014-04-25 11:31:43 +08:00
ad6a273b12
update test url
2014-04-25 11:28:35 +08:00
30af23d003
split monitor to server and client mode #98
2014-04-25 11:25:52 +08:00
ced79630d3
specify jndi and jmx #98
2014-04-25 11:11:15 +08:00
95d3802e77
add formdata support for post request #108
2014-04-24 11:48:58 +08:00
f49bb877c8
clean some code #109
2014-04-24 11:38:13 +08:00
e1aaf1dd11
fix mistake of guava Table #109
2014-04-24 11:05:49 +08:00
8ba2da146c
request method #108 and more cookie #109 config
2014-04-24 10:51:37 +08:00
b06aa489fb
[BugFix]Only one url from sourceRegion can be extracted #107
2014-04-18 17:48:26 +08:00
08fa3b01c1
when download error, throw an exception instead of calling onError and returning peacefully. #105
2014-04-17 17:53:12 +08:00
27b37e8164
extension point and sample for JMX support #98
2014-04-17 08:12:37 +08:00
a5db6cf292
some monitor and JMX support #98
2014-04-17 00:35:09 +08:00
f39aa435cf
add null check #104
2014-04-16 19:46:32 +08:00
42bbe40a37
[Bugfix]Urls will be lost when call setScheduler() #104
2014-04-16 19:45:17 +08:00
163773af6b
combine two try-catch block into one, make it cleaner.
2014-04-16 16:05:08 +08:00
ec446277b1
some refactor in httpclientdownloader
2014-04-15 15:30:37 +08:00
a03f6a8431
eclipse project
2014-04-15 07:44:43 +08:00
4a035e729a
extension point for LocalDuplicatedRemovedScheduler #95
2014-04-13 23:31:13 +08:00
b249e49748
[Bugfix]loop error when add TargetRequest #99
2014-04-13 23:04:09 +08:00
da2f023c12
Merge pull request #96 from ouyanghuangzheng/master
...
修改了Spider 和site 几处注释
2014-04-13 13:12:12 +08:00
f7950ebcab
fix tests
2014-04-13 13:00:31 +08:00
32ba1b8889
修复几处注释问题
2014-04-13 12:41:15 +08:00
84b897f83b
update AngularJSProcessor
2014-04-13 12:20:57 +08:00
03c251237b
add Json parse support
2014-04-13 10:23:00 +08:00
644e8d1f72
同步官方源码
2014-04-12 22:32:22 +08:00
969ad1766b
change logger style to slf4j for cleaner code
2014-04-06 21:32:20 +08:00
9b2cb43f47
ConfigurablePageProcessor #91
2014-04-05 23:40:10 +08:00
b043ac76d6
change the formatter of log.
...
To use slf4j, we should insert {} into the formatter string.
2014-04-05 11:31:56 +08:00
7aaf837e15
change logger to slf4j style for performance #84
2014-04-04 20:10:00 +08:00
f9b157951d
Merge branch 'master' of github.com:code4craft/webmagic
2014-04-04 20:01:14 +08:00
22c394e629
[doc]
2014-04-04 20:00:58 +08:00
762a3973fd
Modify the log levels of LocalDuplicatedRemovedScheduler.java
...
The old version will print a debug level log each time the push method is
called. So sometimes, when a html page have multiple links for the same
page, the debug log will appears more than once. Also, when we meet a
duplicate URL, it will also print a log, which will be confusing.
I change the level of it to trace. And each time a URL is really push into
queue, print a debug level log.
2014-04-04 15:53:46 +08:00
a1c7e826f7
fix dep of slf4j-log4j12
2014-04-03 23:04:31 +08:00
01848301d4
encode illegal charactors in url #80
2014-04-01 22:14:30 +08:00
2780423e60
enable blank space in quotes in UrlUtils.fixAllRelativeHrefs #80
2014-04-01 20:35:11 +08:00
97b6f46280
Bugfix: break loop in addTargetRequests #81
2014-04-01 20:12:25 +08:00
8d8194bee4
Change HashMap to LinkedHashMap in ResultItems for same order of input and output #76
2014-03-25 08:23:20 +08:00
8b35d79569
Do not cache document in Selectable for selected Html element #73
2014-03-19 22:19:06 +08:00
6201fd6966
add worker as container
2014-03-17 23:01:58 +08:00
6c11718566
Clean project structure #70
2014-03-14 23:24:38 +08:00
9606a173cd
fix ZipCodePageProcessor
2014-03-13 22:55:50 +08:00
4f68368db0
Merge branch 'master' of git.oschina.net:flashsword20/webmagic
...
Conflicts:
webmagic-core/src/main/java/us/codecraft/webmagic/selector/RegexSelector.java
2014-03-13 08:09:37 +08:00
98e2bba099
Merge branch 'master' of github.com:code4craft/webmagic
...
Conflicts:
README.md
pom.xml
webmagic-core/pom.xml
webmagic-extension/pom.xml
webmagic-scripts/pom.xml
2014-03-13 08:07:33 +08:00
757cc9b942
[maven-release-plugin] prepare for next development iteration
2014-03-13 07:49:51 +08:00
63ffb5c792
[maven-release-plugin] prepare release webmaigc-0.4.3
2014-03-13 07:49:27 +08:00
66d4d3c192
Merge branch 'master' into 0.4.x
2014-03-13 07:12:29 +08:00
zwf
yihua.huang
fengwuze
ywooer
Bo LIANG
愤怒的番茄