此前在论坛里面看到wosunmo写的JEECMS采集优化觉得很不错，比官方的方便很多，也更准确，不过也存在些不足，最主要的是不能过滤不想要的内容，比如采集韩寒的博客，发现文章里面有很多重复的内容，查看源码才发现，原来把隐藏的内容也采集进来了。木有办法啊！只有自己试着改改了。代码写的一般，目的只有一个：和大家一起学习，共同进步，把JEECMS整巴适了:）
说明：此次优化基于htmlparser，根据标签名称或者标签属性及属性值 过滤内容，只需要设置，内容地址集及内容参数即可采集，准确率高。
用法：
在“开始HTML”处（没有修改模板，其实现在应该叫“内容区域参数”）写上你要采集的标签名称/标签属性及属性值,如 id=artibody
在“结束HTML”处（没有修改模板，其实现在应该叫“内容区域过滤参数”）写上在内容区域内你要过滤的标签名称/标签属性及属性 值，如：type=text/javascript,class=icon_fx|blkComment otherContent_01
约定：不同标签/属性以逗号(“,”)分隔，相同属性不同值，值与值之间以"|"分隔
1，标签属性/值形式，如：class=articleList|tips,id=fxwb|fxMSN|fxMSN
2，标签名称形式，如：div,p,span
3，混合形式，如：class=articleList|tips,id=fxwb|fxMSN|fxMSN,div,p,span
采集参数样例：
新浪国内新闻:
内容地址集: 开始HTML: class=list_009 结束HTML :空
内容: 开始HTML: id=artibody 结束HTML :type=text/javascript,class=icon_fx|blkComment otherContent_01,style=text-align: right;padding-right:10px;|margin-top:6px;|font-size: 12px ! important;|font-size:12px,id=fxwb|fxMSN|fxMSN|comment_t_show_top,style,script
代码如下：
HTML解析工具类接口：
package com.jeecms.cms.service;
import java.util.List;
import com.jeecms.cms.entity.assist.CmsAcquisition;
/**
* HTML解析工具类接口
* @author javacoo
* @since 2011-10-31
*/
public interface ParseHtmlTool {
/**
* 取得连接集合
* @param orginHtml 原始HTML
* @return 连接集合
*/
List<String> getUrlList( String orginHtml);
/**
* 取得标题集合
* @param orginHtml 原始HTML
* @return 标题集合
*/
List<String> getTitleList(String orginHtml);
/**
* 取得指定区域的HTML内容
* @return 指定区域的HTML内容
*/
String getHtml(String orginHtml);
}
HTML解析工具,HtmlParser实现类：
package com.jeecms.cms.service;
import java.io.BufferedReader;
import java.io.File;
import java.io.FileInputStream;
import java.io.IOException;
import java.io.InputStreamReader;
import java.net.URISyntaxException;
import java.util.ArrayList;
import java.util.HashMap;
import java.util.Iterator;
import java.util.List;
import java.util.Map;
import java.util.regex.Matcher;
import java.util.regex.Pattern;
import org.apache.commons.lang.StringUtils;
import org.htmlparser.Node;
import org.htmlparser.NodeFilter;
import org.htmlparser.Parser;
import org.htmlparser.filters.HasAttributeFilter;
import org.htmlparser.filters.NodeClassFilter;
import org.htmlparser.filters.TagNameFilter;
import org.htmlparser.nodes.RemarkNode;
import org.htmlparser.util.NodeList;
import org.htmlparser.util.ParserException;
import com.jeecms.cms.entity.assist.CmsAcquisition;
/**
* HTML解析工具,HtmlParser实现类
* @author javacoo
* @since 2011-10-31
*/
public class HtmlParserImpl implements ParseHtmlTool{
/**连接集合标志*/
private static String LINK_LIST = "linkList";
/**标题集合标志*/
private static String TITLE_LIST = "titleList";
/**单标签标志*/
private static String SINGLE_TAG = "singleTag";
/**连接正则表达式*/
private static String LINK_REGX = "<a.*href=\"(.*?)\".*>(.*?)</a>";
/**正则表达式对象*/
private Pattern pt = Pattern.compile(LINK_REGX);
/**采集参数bean*/
private ParamBean paramBean;
public HtmlParserImpl(CmsAcquisition acqu){
parseRequestParam(acqu);
}
/**
* 取得标题集合
* @param orginHtml 原始HTML
* @return 标题集合
*/
public List<String> getTitleList(String orginHtml) {
orginHtml = getHtmlByFilter(paramBean.getLinksetStartMap(), paramBean.getLinksetEndMap(),orginHtml);
if (StringUtils.isNotEmpty(orginHtml)) {
return getUrlOrTitleListByType(orginHtml,TITLE_LIST);
}
return null;
}
/**
* 取得连接集合
* @param orginHtml 原始HTML
* @return 连接集合
*/
public List<String> getUrlList(String orginHtml) {
orginHtml = getHtmlByFilter(paramBean.getLinksetStartMap(), paramBean.getLinksetEndMap(),orginHtml);
if (StringUtils.isNotEmpty(orginHtml)) {
return getUrlOrTitleListByType(orginHtml,LINK_LIST);
}
return null;
}
/**
* 取得指定区域的HTML内容
* @param orginHtml 原始HTML
* @return 指定区域的HTML内容
* @throws ParserException
*/
public String getHtml(String orginHtml) {
orginHtml = getHtmlByFilter(paramBean.getContentStartMap(), paramBean.getContentEndMap(),orginHtml);
return orginHtml;
}
/**
* 解析采集参数,并封装到ParamBean
* @param acqu 原始采集参数
* @return 采集参数封装bean
*/
private void parseRequestParam(CmsAcquisition acqu){
paramBean = new ParamBean();
if(!StringUtils.isEmpty(acqu.getLinksetStart())){
paramBean.setLinksetStartMap(populateParamMap(acqu.getLinksetStart()));
}
if(!StringUtils.isEmpty(acqu.getLinksetEnd())){
paramBean.setLinksetEndMap(populateParamMap(acqu.getLinksetEnd()));
}
if(!StringUtils.isEmpty(acqu.getContentStart())){
paramBean.setContentStartMap(populateParamMap(acqu.getContentStart()));
}
if(!StringUtils.isEmpty(acqu.getContentEnd())){
paramBean.setContentEndMap(populateParamMap(acqu.getContentEnd()));
}
}
/**
* 得到地址集
* @param html html内容
* @param type 1 ：取得连接集合，2：取得标题集合
* @return 连接或者标题集合
*/
private List<String> getUrlOrTitleListByType(String html, String type) {
List<String> resultList = new ArrayList<String>();
Matcher m = pt.matcher(html);
String result = "";
int pos = 1;
if(TITLE_LIST.equals(type)){
pos = 2;
}
while (m.find()) {
result = m.group(pos);
resultList.add(result);
}
return resultList;
}
/**
* 取得指定区域的HTML内容
* @param tagMap 标签MAP
* @param removeTagMap 要过滤的标签MAP
* @param orginHtml 原始HTML
* @return 指定区域的HTML内容
* @throws ParserException
*/
private String getHtmlByFilter(Map<String, String> tagMap,
Map<String, String> removeTagMap, String orginHtml) {
try {
Parser parser = new Parser();
parser.setInputHTML(orginHtml);
// 第一步取得指定属性/标签内容
String tempKey = null;
String tempValue = null;
String[] tempValueArr = null;
StringBuilder sb = new StringBuilder();
NodeFilter filter = null;
for(Iterator<String> it = tagMap.keySet().iterator(); it.hasNext();){
tempKey = it.next();
tempValue = tagMap.get(tempKey);
if(tempValue.contains("|")){
tempValueArr = tempValue.split("\\|");
}else{
tempValueArr = new String[]{tempValue};
}
for(String value : tempValueArr){
filter = populateFilter(tempKey,value);
appendHtmlByFilter(parser, filter, sb);
}
}
// 第二步过滤指定属性/标签内容
String contentHtml = sb.toString();
for (Iterator<String> it = removeTagMap.keySet().iterator(); it
.hasNext();) {
tempKey = it.next();
tempValue = removeTagMap.get(tempKey);
if(tempValue.contains("|")){
tempValueArr = tempValue.split("\\|");
}else{
tempValueArr = new String[]{tempValue};
}
for(String value : tempValueArr){
filter = populateFilter(tempKey,value);
contentHtml = removeHtmlByFilter(parser, filter, contentHtml);
}
}
//第三步过滤注释
filter = new NodeClassFilter(RemarkNode.class);
contentHtml = removeHtmlByFilter(parser, filter, contentHtml);
System.out.println("=================================结果=======================================");
System.out.println(contentHtml);
return contentHtml;
} catch (ParserException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
return "";
}
/**
* 解析并组装采集参数，支持标签属性/值形式和标签名称形式，可混合使用
* <li>约定采集参数格式如下</li>
* <li>1，标签属性/值形式，如：class=articleList|tips,id=fxwb|fxMSN|fxMSN</li>
* <li>2，标签名称形式，如：div,p,span</li>
* <li>3，混合形式，如：class=articleList|tips,id=fxwb|fxMSN|fxMSN,div,p,span</li>
* @param paramStr 参数字符串
*/
private Map<String, String> populateParamMap(String paramStr) {
Map<String, String> paramMap = new HashMap<String, String>();
String[] paramStrArr = paramStr.split(",");
String[] tempStrArr = null;
StringBuilder sb = new StringBuilder();
for(String temp : paramStrArr){
if(temp.contains("=")){
tempStrArr = temp.split("=");
paramMap.put(tempStrArr[0], tempStrArr[1]);
}else{
if(StringUtils.isNotEmpty(temp)){
sb.append(temp).append("|");
}
}
}
if(StringUtils.isNotEmpty(sb.toString())){
paramMap.put(SINGLE_TAG, sb.substring(0, sb.length() - 1));
}
return paramMap;
}
/**
* 组装过滤器
* @param key 键
* @param value 值
* @return 过滤器
*/
private NodeFilter populateFilter(String key,String value) {
NodeFilter filter;
if(SINGLE_TAG.equals(key)){
filter = new TagNameFilter(value);
}else{
filter = new HasAttributeFilter(key,value);
}
return filter;
}
/**
* 过滤指定属性标签HTML
* @param parser 解析器
* @param filter 属性过滤器
* @param orginHtml 原始HTML
* @return 过滤后HTML
* @throws ParserException
*/
private String removeHtmlByFilter(Parser parser, NodeFilter filter,String orginHtml) throws ParserException {
parser.setInputHTML(orginHtml);
NodeList nodes = parser.extractAllNodesThatMatch(filter);
for (int i = 0; i < nodes.size(); i++) {
Node textnode = (Node) nodes.elementAt(i);
orginHtml = StringUtils.remove(orginHtml, textnode.toHtml());
}
return orginHtml;
}
/**
* 取得所有指定属性/标签的HTML
* @param parser 解析器
* @param filter 过滤器
* @param sb
* @throws ParserException
*/
private void appendHtmlByFilter(Parser parser, NodeFilter filter,
StringBuilder sb) throws ParserException {
NodeList nodes = parser.extractAllNodesThatMatch(filter);
System.out.println("匹配节点数:" + nodes.size());
for (int i = 0; i < nodes.size(); i++) {
Node textnode = (Node) nodes.elementAt(i);
sb.append(textnode.toHtml());
}
}
/**
* 解析并组装采集参数，支持标签属性/值形式和标签名称形式，可混合使用
* <li>约定采集参数格式如下</li>
* <li>1，标签属性/值形式，如：class=articleList|tips,id=fxwb|fxMSN|fxMSN</li>
* <li>2，标签名称形式，如：div,p,span</li>
* <li>3，混合形式，如：class=articleList|tips,id=fxwb|fxMSN|fxMSN,div,p,span</li>
* @param paramMap 参数map
* @param str 参数字符串
*/
private void populateParamMap(Map<String, String> paramMap,String paramStr) {
String[] paramStrArr = paramStr.split(",");
String[] tempStrArr = null;
StringBuilder sb = new StringBuilder();
for(String temp : paramStrArr){
if(temp.contains("=")){
tempStrArr = temp.split("=");
paramMap.put(tempStrArr[0], tempStrArr[1]);
}else{
if(StringUtils.isNotEmpty(temp)){
sb.append(temp).append("|");
}
}
}
if(StringUtils.isNotEmpty(sb.toString())){
paramMap.put(SINGLE_TAG, sb.substring(0, sb.length() - 1));
}
}
/**
* 测试方法-打开文件并返回内容
* @param szFileName 文件绝对地址
* @param charset 字符集
* @return 内容
*/
public static String openFile(String szFileName,String charset) {
try {
BufferedReader bis = new BufferedReader(new InputStreamReader(
new FileInputStream(new File(szFileName)), charset));
StringBuilder szContent = new StringBuilder();
String szTemp;
while ((szTemp = bis.readLine()) != null) {
szContent.append(szTemp).append("\n");
}
bis.close();
return szContent.toString();
} catch (Exception e) {
return "";
}
}
/**
* 测试取得连接地址和标题
* @throws ParserException
*/
public void testFetchLinkAndTitle() throws ParserException{
String html = openFile("F:\\4.htm","UTF-8");
String result = "";
Map<String, String> map = new HashMap<String, String>();
map.put("class", "m_list");
Map<String, String> notMap = new HashMap<String, String>();
//notMap.put("class", "atc_ic_f");
result = getHtmlByFilter(map,notMap,html);
System.out.println("=============================result============================");
System.out.println(result);
System.out.println("==========================================================");
Pattern pt = Pattern.compile("<a.*href=\"(.*?)\".*>(.*?)</a>");
Matcher m = pt.matcher(result);
String link = null;
String title = null;
while (m.find()) {
link = m.group(1);
title = m.group(2);
if (StringUtils.isNotEmpty(link)) {
System.out.println("url : " + link);
System.out.println("title : " + title);
}
}
}
/**
* 测试取得内容
* @throws ParserException
*/
public void testFetchContent() throws ParserException{
String html = openFile("F:\\6.shtml","GB2312");
Map<String, String> map = new HashMap<String, String>();
map.put("id", "artibody");
Map<String, String> notMap = new HashMap<String, String>();
notMap.put(SINGLE_TAG, "style|script");
notMap.put("type", "text/javascript");
notMap.put("class", "icon_fx|blkComment otherContent_01");
notMap.put("style", "text-align: right;padding-right:10px;|margin-top:6px;|font-size: 12px ! important;|font-size:12px");
notMap.put("id", "fxwb|fxMSN|fxMSN|comment_t_show_top");
getHtmlByFilter(map,notMap,html);
}
/**
* 测试解析参数
*/
public void testParseParam(){
Map<String, String> map = new HashMap<String, String>();
populateParamMap(map,"class=articleList|tips,p,div");
String tempKey = null;
String tempValue = null;
String[] tempValueArr = null;
for (Iterator<String> it = map.keySet().iterator(); it.hasNext();) {
tempKey = it.next();
tempValue = map.get(tempKey);
if(tempValue.contains("|")){
tempValueArr = tempValue.split("\\|");
}else{
tempValueArr = new String[]{tempValue};
}
for(String value : tempValueArr){
System.out.println("tempKey:" + tempKey);
System.out.println("value:" + value);
}
}
}
/**
* 测试过滤标签
* @throws ParserException
*/
public void testRemarkFilter() throws ParserException{
String html = openFile("F:\\6.shtml","GB2312");
System.out.println("=========================过滤注释前HTML==================================");
System.out.println(html);
NodeFilter filter = new NodeClassFilter(RemarkNode.class);
html = removeHtmlByFilter(new Parser(), filter, html);
System.out.println("=========================过滤注释后HTML==================================");
System.out.println(html);
}
public static void main(String[] args) throws ParserException,
URISyntaxException, IOException {
HtmlParserImpl parseHtmlTool = new HtmlParserImpl(new CmsAcquisition());
//parseHtmlTool.testParseParam();
//parseHtmlTool.testFetchLinkAndTitle();
//parseHtmlTool.testFetchContent();
//parseHtmlTool.testRemarkFilter();
}
}
采集参数封装bean：
package com.jeecms.cms.service;
import java.util.HashMap;
import java.util.Map;
/**
* 采集参数封装bean
* @author javacoo
* @since 2011-10-31
*/
public class ParamBean {
/**待采集连接区域属性MAP*/
private Map<String, String> linksetStartMap = new HashMap<String, String>();
/**待采集连接区域过滤属性MAP*/
private Map<String, String> linksetEndMap = new HashMap<String, String>();
/**待采集内容区域属性MAP*/
private Map<String, String> contentStartMap = new HashMap<String, String>();
/**待采集内容区域过滤属性MAP*/
private Map<String, String> contentEndMap = new HashMap<String, String>();
public Map<String, String> getLinksetStartMap() {
return linksetStartMap;
}
public void setLinksetStartMap(Map<String, String> linksetStartMap) {
this.linksetStartMap = linksetStartMap;
}
public Map<String, String> getLinksetEndMap() {
return linksetEndMap;
}
public void setLinksetEndMap(Map<String, String> linksetEndMap) {
this.linksetEndMap = linksetEndMap;
}
public Map<String, String> getContentStartMap() {
return contentStartMap;
}
public void setContentStartMap(Map<String, String> contentStartMap) {
this.contentStartMap = contentStartMap;
}
public Map<String, String> getContentEndMap() {
return contentEndMap;
}
public void setContentEndMap(Map<String, String> contentEndMap) {
this.contentEndMap = contentEndMap;
}
}
AcquisitionSvcImpl类
package com.jeecms.cms.service;
import java.io.IOException;
import java.net.URI;
import java.util.List;
import org.apache.commons.lang.StringUtils;
import org.apache.http.HttpEntity;
import org.apache.http.HttpHost;
import org.apache.http.HttpResponse;
import org.apache.http.StatusLine;
import org.apache.http.client.ClientProtocolException;
import org.apache.http.client.HttpClient;
import org.apache.http.client.HttpResponseException;
import org.apache.http.client.ResponseHandler;
import org.apache.http.client.methods.HttpGet;
import org.apache.http.conn.params.ConnRoutePNames;
import org.apache.http.impl.client.DefaultHttpClient;
import org.apache.http.util.EntityUtils;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.stereotype.Service;
import com.jeecms.cms.entity.assist.CmsAcquisition;
import com.jeecms.cms.entity.main.Content;
import com.jeecms.cms.manager.assist.CmsAcquisitionMng;
@Service
public class AcquisitionSvcImpl implements AcquisitionSvc {
private Logger log = LoggerFactory.getLogger(AcquisitionSvcImpl.class);
public boolean start(Integer id) {
CmsAcquisition acqu = cmsAcquisitionMng.findById(id);
if (acqu == null || acqu.getStatus() == CmsAcquisition.START) {
return false;
}
Thread thread = new AcquisitionThread(acqu);
thread.start();
return true;
}
private CmsAcquisitionMng cmsAcquisitionMng;
@Autowired
public void setCmsAcquisitionMng(CmsAcquisitionMng cmsAcquisitionMng) {
this.cmsAcquisitionMng = cmsAcquisitionMng;
}
private class AcquisitionThread extends Thread {
private CmsAcquisition acqu;
/**HTML解析工具类*/
private ParseHtmlTool parseHtmlTool;
public AcquisitionThread(CmsAcquisition acqu) {
super(acqu.getClass().getName() + "#" + acqu.getId());
this.acqu = acqu;
parseHtmlTool = new HtmlParserImpl(acqu);
}
@Override
public void run() {
if (acqu == null) {
return;
}
long tStart = System.currentTimeMillis();
System.out.println(Thread.currentThread().getName() + "开始...");
acqu = cmsAcquisitionMng.start(acqu.getId());
String[] plans = acqu.getAllPlans();
//HttpHost proxy = new HttpHost("128.160.64.5", 1235);
HttpClient client = new DefaultHttpClient();
//client.getParams().setParameter(ConnRoutePNames.DEFAULT_PROXY,proxy);
CharsetHandler handler = new CharsetHandler(acqu.getPageEncoding());
List<String> urlList = null;
List<String> titleList = null;
String url = null;
String html = null;
int currNum = acqu.getCurrNum();
int currItem = acqu.getCurrItem();
Integer acquId = acqu.getId();
try {
for (int i = plans.length - currNum; i >= 0; i--) {
url = plans[i];
HttpGet httpget = new HttpGet(new URI(url.trim()));
html = client.execute(httpget, handler);
urlList = parseHtmlTool.getUrlList(html);
titleList = parseHtmlTool.getTitleList(html);
if (urlList != null) {
for (int j = urlList.size() - currItem; j >= 0; j--) {
if (cmsAcquisitionMng.isNeedBreak(acqu.getId(),
plans.length - i, urlList.size() - j,
urlList.size())) {
client.getConnectionManager().shutdown();
log.info("Acquisition#{} breaked", acqu.getId());
return;
}
if (acqu.getPauseTime() > 0) {
try {
Thread.sleep(acqu.getPauseTime());
} catch (InterruptedException e) {
log.warn("", e);
}
}
saveContent(client, handler, acquId, urlList.get(j), titleList.get(j));
}
}
currItem = 1;
}
} catch (Exception e) {
e.printStackTrace();
log.warn(null, e);
}
client.getConnectionManager().shutdown();
cmsAcquisitionMng.end(acqu.getId());
System.out.println(Thread.currentThread().getName() + "结束.");
long tEnd = System.currentTimeMillis();
System.out.println("总共用时:"+ (tEnd - tStart) + "millions");
log.info("Acquisition#{} complete", acqu.getId());
}
private Content saveContent(HttpClient client, CharsetHandler handler,Integer acquId, String url, String title) {
try {
HttpGet httpget = new HttpGet(new URI(url.trim()));;
String html = client.execute(httpget, handler);
String txt = parseHtmlTool.getHtml(html);
return cmsAcquisitionMng.saveContent(title, txt,acquId);
} catch (Exception e) {
log.warn(null, e);
e.printStackTrace();
return null;
}
}
}
private class CharsetHandler implements ResponseHandler<String> {
private String charset;
public CharsetHandler(String charset) {
this.charset = charset;
}
public String handleResponse(HttpResponse response)
throws ClientProtocolException, IOException {
StatusLine statusLine = response.getStatusLine();
if (statusLine.getStatusCode() >= 300) {
throw new HttpResponseException(statusLine.getStatusCode(),
statusLine.getReasonPhrase());
}
HttpEntity entity = response.getEntity();
if (entity != null) {
if (!StringUtils.isBlank(charset)) {
return EntityUtils.toString(entity, charset);
} else {
return EntityUtils.toString(entity);
}
} else {
return null;
}
}
}
}
采集结果：
<div class="blkContainerSblkCon" id="artibody">
<p> 四川新闻网自贡10月28日讯 (记者 张贤明 黄国苗) 10月28日上午，中国***自贡市第十一届委员会第一次全体会议选举产生新一届自贡市委常委。</p>
<p> 会议以无记名投票方式，选举雷洪金为市委书记，彭琳、谭豹为市委副书记；向华全、陈吉明、姜怡、张乾华、李宪虎、曹俊杰、杨家禄、吴丕当选市委常委。</p>
<p> 在自贡市纪委十一届一次全体会议上，张乾华当选为新一届自贡市纪委书记。</p> <div style="clear:both;height:0;visibility:hiddden;overflow:hidden;"></div>
</div>
继续优化：
还有很多木有想到的情况，希望大家一起来把JEECMS的采集功能做的更完善，更方便，更快捷，更强大！！！
|
|
