Release 1.0.0 (23 Sep 2020)

Key Features

This release focused on making improvements in 3 main areas: Performance, Security, and Usability.

  • Performance

    New enhancements were made to the engine to further improve the performance of ad-hoc interactive queries. These enhancements include changes to the dynamic filter feature to use a more efficient implementation of bloom filters, as well as optimizing the dynamic filter source operator so that dynamic filters can be collected and used as soon as possible. Other performance optimizations include the introduction of predicate pushdown to allow OR predicates to be pushed to the ORC reader, as well as supporting OR predicates during split filtering with the heuristic index feature. Finally, auto compaction was introduced for ORC files to reduce the number of ORC files after insert and update operations.

  • Security

    In this release the community focused on further securing the query engine. Several vulnerabilities were addressed in the code, and an audit log was introduced. Furthermore, a new feature was introduced to allow administrators to encrypt sensitive information like data source credentials found in catalog property files.

  • Usability

    Usability was another area of main focus for this release. In order to provide a database-like user experience, the community introduced a migration tool that helps migrate SQL queries from other engines to work in openLooKeng. Other usability enhancements include capturing ORC cache metrics via JMX to allow administrators to view cache hits and misses, and also improving the CACHE sql command to provide more flexibility as to what can be cached. Furthermore, changes were made to installation and deployment scripts to support deployments that contain a mix of ARM and x86 nodes. New scripts are introduced to allow administrators to deploy openLooKeng service based on containers.

AreaFeaturePR #s
SecurityAudit Logging160
Security Vulnerability Fixes113
Handle CVEs144
Installation DeploymentScripts to help deploy and run openLooKeng on Docker & Kubernetes50
Support installation on nodes with mixed architectures of ARM vs. x86.5
ToolsSupport migration of Impala SQL to ANSI SQL52
Usability enhancements to SQL migration tool to help user to easily identify differences between SQL statements, and accept changes.159
ODBC DriverSupport openLooKeng complex data type such as ARRAY, JSON,MAP, and ROW.11 (in hetu-odbc-gateway’s repo) & 6 in (hetu-odbc-driver’s repo)
Bidirectional Scrollable cursors supported so as to enable easier iteration on tables & result sets. Position also can be specified to set the cursor to10
Heuristic IndexRefactor heuristic indexer, change to openLooKeng plugin, use HetuFileSystem Client40
Add additional configs to give admin better control of the index cache. For, example, admin can now set the maximum amount of memory the index cache can take.85
Support split filtering at schedule time for queries that contain OR predicate.93
Add session property to disable heuristic index while the server is running.126
OptimizersPush predicates such as OR predicates to ORC reader to perform data skipping.103
Execution PipelineSeveral Dynamic filter enhancements such as the use of an internal bloom filter implementation that gives better performance, as well as improved row filtering logic, and added State Store listeners to collect newly merged dynamic filters.65, 86, 100, 112, 137
Enhanced dynamic filters to support more types of blocks, and enabled block level filtering to take advantage of vectorization.101, 117
ORC IUD/ACIDSupported threshold based auto triggering and handling of compaction so that user intervention is not required and the query performance doesn’t degrade when mutation operations are performed on the table (Currently supported for Hive)141
Auto cleanup of ORC files after compaction operation.89
Catalog ManagementEncryption of sensitive information like passwords in catalog properties file.170
DC ConnectorImproved the performance of DC connector by supporting Metadata & plan cache.128
ORC Data CacheCapture ORC cache metrics via JMX.33, 142
CACHE sql command usability enhancements to provide more flexibility as to what partitions can be cached.90, 99

Obtaining the Document

For details, see https://gitee.com/openlookeng/hetu-core/tree/1.0.0/hetu-docs/en

有奖捉虫

“有虫”文档片段

0/500

存在的问题

文档存在风险与错误

● 拼写,格式,无效链接等错误;

● 技术原理、功能、规格等描述和软件不一致,存在错误;

● 原理图、架构图等存在错误;

● 版本号不匹配:文档版本或内容描述和实际软件不一致;

● 对重要数据或系统存在风险的操作,缺少安全提示;

● 排版不美观,影响阅读;

内容描述不清晰

● 描述存在歧义;

● 图形、表格、文字等晦涩难懂;

● 逻辑不清晰,该分类、分项、分步骤的没有给出;

内容获取有困难

● 很难通过搜索引擎,openLooKeng官网,相关博客找到所需内容;

示例代码有错误

● 命令、命令参数等错误;

● 命令无法执行或无法完成对应功能;

内容有缺失

● 关键步骤错误或缺失,无法指导用户完成任务,比如安装、配置、部署等;

● 逻辑不清晰,该分类、分项、分步骤的没有给出

● 图形、表格、文字等晦涩难懂

● 缺少必要的前提条件、注意事项等;

● 描述存在歧义

0/500

您对文档的总体满意度

非常不满意
非常满意

请问是什么原因让您参与到这个问题中

您的邮箱

创Issue赢奖品
根据您的反馈,会自动生成issue模板。您只需点击按钮,创建issue即可。
有奖捉虫