最近着手处理大批量数据的任务。
现状是这样的,一个数据采集程序承载大批量数据的存储和检索。后期可能需要对大批量数据进行统计。
数据分布情况
13个点定时生成采集结果到4个文件(小文件生成周期是5分钟)
名称 大小(b) gather_1_2014-02-27-14-50-0.txt 568497 gather_1_2014-02-27-14-50-1.txt 568665 gather_1_2014-02-27-14-50-2.txt 568172 gather_1_2014-02-27-14-50-3.txt 568275
同步使用shell脚本对四个文件入到sybase_iq库的一张表tab_tmp_2014_2_27中.
每天数据量大概是3亿条,所以小文件的总量大概是3G。小文件数量大,单表容量大执行复合主键查询,由原来2s延时变成了,5~10分钟。
针对上述情况需要对目前的储存结构进行优化。
才是看了下相关系统 catior使用的是环状数据库,存储相关的数据优点方便生成MRTG图,缺点不利于数据统计。后来引入elasticsearch来对大数据检索进行优化。
测试平台
cpu: AMD Opteron(tm) Processor 6136 64bit 2.4GHz * 32 内存: 64G 硬盘:1.5T 操作系统:Red Hat Enterprise Linux Server release 6.4 (Santiago)
读取文件的目录结构:
[test@test001 data]$ ls 0 1 2 3
简单测试代码:
public class FileReader { private File file; private String splitCharactor; private Map<String, Class<?>> colNames; private static final Logger LOG = Logger.getLogger(FileReader.class); /** * @param path * 文件路径 * @param fileName * 文件名 * @param splitCharactor * 拆分字符 * @param colNames * 主键名称 */ public FileReader(File file, String splitCharactor, Map<String, Class<?>> colNames) { this.file = file; this.splitCharactor = splitCharactor; this.colNames = colNames; } /** * 读取文件 * * @return * @throws Exception */ public List<Map<String, Object>> readFile() throws Exception { List<Map<String, Object>> list = new ArrayList<Map<String, Object>>(); if (!file.isFile()) { throw new Exception("File not exists." + file.getName()); } LineIterator lineIterator = null; try { lineIterator = FileUtils.lineIterator(file, "UTF-8"); while (lineIterator.hasNext()) { String line = lineIterator.next(); String[] values = line.split(splitCharactor); if (colNames.size() != values.length) { continue; } Map<String, Object> map = new HashMap<String, Object>(); Iterator<Entry<String, Class<?>>> iterator = colNames.entrySet() .iterator(); int count = 0; while (iterator.hasNext()) { Entry<String, Class<?>> entry = iterator.next(); Object value = values[count]; if (!String.class.equals(entry.getValue())) { value = entry.getValue().getMethod("valueOf", String.class) .invoke(null, value); } map.put(entry.getKey(), value); count++; } list.add(map); } } catch (IOException e) { LOG.error("File reading line error." + e.toString(), e); } finally { LineIterator.closeQuietly(lineIterator); } return list; } }
public class StreamIntoEs { public static class ChildThread extends Thread { int number; public ChildThread(int number) { this.number = number; } @Override public void run() { Settings settings = ImmutableSettings.settingsBuilder() .put("client.transport.sniff", true) .put("client.transport.ping_timeout", 100) .put("cluster.name", "elasticsearch").build(); TransportClient client = new TransportClient(settings) .addTransportAddress(new InetSocketTransportAddress("192.168.32.228", 9300)); File dir = new File("/export/home/es/data/" + number); LinkedHashMap<String, Class<?>> colNames = new LinkedHashMap<String, Class<?>>(); colNames.put("aa", Long.class); colNames.put("bb", String.class); colNames.put("cc", String.class); colNames.put("dd", Integer.class); colNames.put("ee", Long.class); colNames.put("ff", Long.class); colNames.put("hh", Long.class); int count = 0; long startTime = System.currentTimeMillis(); for (File file : dir.listFiles()) { int currentCount = 0; long startCurrentTime = System.currentTimeMillis(); FileReader reader = new FileReader(file, "\\$", colNames); BulkResponse resp = null; BulkRequestBuilder bulkRequest = client.prepareBulk(); try { List<Map<String, Object>> results = reader.readFile(); for (Map<String, Object> col : results) { bulkRequest.add(client.prepareIndex("flux", "fluxdata") .setSource(JSON.toJSONString(col)).setId(col.get("getway")+"##"+col.get("port_info")+"##"+col.get("device_id")+"##"+col.get("collecttime"))); count++; currentCount++; } resp = bulkRequest.execute().actionGet(); } catch (Exception e) { // TODO Auto-generated catch block e.printStackTrace(); } long endCurrentTime = System.currentTimeMillis(); System.out.println("[thread-" + number + "-]per count:" + currentCount); System.out.println("[thread-" + number + "-]per time:" + (endCurrentTime - startCurrentTime)); System.out.println("[thread-" + number + "-]per count/s:" + (float) currentCount / (endCurrentTime - startCurrentTime) * 1000); System.out.println("[thread-" + number + "-]per count/s:" + resp.toString()); } long endTime = System.currentTimeMillis(); System.out.println("[thread-" + number + "-]total count:" + count); System.out.println("[thread-" + number + "-]total time:" + (endTime - startTime)); System.out.println("[thread-" + number + "-]total count/s:" + (float) count / (endTime - startTime) * 1000); // IndexRequest request = // = client.index(request); } } public static void main(String args[]) { for (int i = 0; i < 4; i++) { ChildThread childThread = new ChildThread(i); childThread.start(); } } }
起了4个线程来做入库,每个文件解析完成进行一次批处理。
初始化脚本:
curl -XDELETE 'http://192.168.32.228:9200/twitter/'
curl -XPUT 'http://192.168.32.228:9200/twitter/' -d ' { "index" :{ "number_of_shards" : 5, "number_of_replicas ": 0, "index.refresh_interval": "-1", "index.translog.flush_threshold_ops": "100000" } }'
curl -XPUT 'http://192.168.32.228:9200/twiter/twiterdata/_mapping' -d ' { "twiterdata": { "aa" : {"type" : "long", "index" : "not_analyzed"}, "bb" : {"type" : "String", "index" : "not_analyzed"}, "cc" : {"type" : "String", "index" : "not_analyzed"}, "dd" : {"type" : "integer", "index" : "not_analyzed"}, "ee" : {"type" : "long", "index" : "no"}, "ff" : {"type" : "long", "index" : "no"}, "gg" : {"type" : "long", "index" : "no"}, "hh" : {"type" : "long", "index" : "no"}, "ii" : {"type" : "long", "index" : "no"}, "jj" : {"type" : "long", "index" : "no"}, "kk" : {"type" : "long", "index" : "no"}, } }
执行效率参考:
不开启refresh_interval [test@test001 bin]$ more StreamIntoEs.out|grep total [thread-2-]total count:1199411 [thread-2-]total time:1223718 [thread-2-]total count/s:980.1368 [thread-1-]total count:1447214 [thread-1-]total time:1393528 [thread-1-]total count/s:1038.5253 [thread-0-]total count:1508043 [thread-0-]total time:1430167 [thread-0-]total count/s:1054.4524 [thread-3-]total count:1650576 [thread-3-]total time:1471103 [thread-3-]total count/s:1121.9989 4195.1134 开启refresh_interval [test@test001 bin]$ more StreamIntoEs.out |grep total [thread-2-]total count:1199411 [thread-2-]total time:996111 [thread-2-]total count/s:1204.0938 [thread-1-]total count:1447214 [thread-1-]total time:1163207 [thread-1-]total count/s:1244.1586 [thread-0-]total count:1508043 [thread-0-]total time:1202682 [thread-0-]total count/s:1253.9 [thread-3-]total count:1650576 [thread-3-]total time:1236239 [thread-3-]total count/s:1335.1593 5037.3117 开启refresh_interval 字段类型转换 [test@test001 bin]$ more StreamIntoEs.out |grep total [thread-2-]total count:1199411 [thread-2-]total time:1065229 [thread-2-]total count/s:1125.9653 [thread-1-]total count:1447214 [thread-1-]total time:1218342 [thread-1-]total count/s:1187.8552 [thread-0-]total count:1508043 [thread-0-]total time:1230474 [thread-0-]total count/s:1225.5789 [thread-3-]total count:1650576 [thread-3-]total time:1274027 [thread-3-]total count/s:1295.5581 4834.9575 开启refresh_interval 字段类型转换 设置id [thread-2-]total count:1199411 [thread-2-]total time:912251 [thread-2-]total count/s:1314.7817 [thread-1-]total count:1447214 [thread-1-]total time:1067117 [thread-1-]total count/s:1356.1906 [thread-0-]total count:1508043 [thread-0-]total time:1090577 [thread-0-]total count/s:1382.7937 [thread-3-]total count:1650576 [thread-3-]total time:1128490 [thread-3-]total count/s:1462.6412 5516.4072
580M的数据平均用时大概是20分钟。索引文件大约为1.76G
相关测试结果可以参考这里:
elasticsearch 性能测试
相关推荐
Linux环境下使用sqlplus工具将oracle中的数据导入到elasticsearch中。只需要在es_bulk_tool.properties配置sql即可实现数据的批量导入。在elasticsearch6中测试通过。shell脚本需要使用sqlplus。
ElasticSearch数据导出 elasticsearch单文档数据导出 支持自定义查询 导出数据Json文件
ElasticSearch官方测试数据
消费kafka数据,然后批量导入到Elasticsearch,本例子使用的kafka版本0.10,es版本是6.4,使用bulk方式批量导入到es中,也可以一条一条的导入,不过比较慢。 <groupId>org.elasticsearch <artifactId>elastic...
因为你不知道将Hive的数据导入到了ElasticSearch后,数据量是否准确,所以需要钉钉报警校验ElasticSearch和Hive数据仓库内的数据质量,注意,这个项目打包后,最好另起一个进程调用,并且开始时间为文章1或者2最大...
ES 官方示例数据
项目实战:Java一站式解决Hive内用Spark取数,新建ES索引,灌入数据,并且采用ES别名机制,实现ES数据更新的无缝更新,底层采用Spark计算框架,数据较快。
elastic数据导出工具,可以直接将es中的数据导入存储到Mysql,也可以指定存储到本地文件夹中,可以选择以json格式或者sql语句的形式
ElasticSearch测试数据
springboot整合elasticsearch7,进行数据同步。elasticsearch相关度查询、排序。高亮显示;自动补全等功能。代码仅供参考,代码中有具体的注释,可以根据代码及注释内容,对自己项目架构及业务进行修改、整合。
好记性不如烂笔头哦~,ElasticSearch,简称es,es是一个开源的高拓展的分布式全文搜索引擎它可以近乎实时的存储、检索数据;本身拓展性很好,可以拓展到上百台服务器,处理PB级别的数据。es也是用Java开发并使用...
真实开发环境的es设计开发文档,可以参考试试,版本比较老了
可用于Elasticsearch学习使用的测试数据。数据格式为xml,需要通过logstash导入Elasticsearch。
elasticsearch-7.17.6及对应版本IK分词 适合人群:elasticsearch初学者 Elasticsearch 是位于 Elastic Stack 核心的分布式搜索...随着您的数据和查询量的增长,Elasticsearch 的分布式特性使您的部署能够随之无缝增长
Elasticsearch官方提供数据案例account.json
该工具支持将arcgis的shp数据导入到ES中,支持多面数据或带洞的面数据
ElasticSearch数据导入 文件导入 单个索引导入 文件按格式采用Json
Elasticsearch 是一个实时的分布式搜索分析引擎, 它能让你以一个之前从未有过的速度和规模,去探索你的数据。 它被用作全文检索、结构化搜索、分析以及这三个功能的组合: Wikipedia 使用 Elasticsearch 提供带有...
elasticsearch elasticsearch-6.2.2 elasticsearch-6.2.2.zip 下载