VDM Connector

There are use cases that multiple data sources need to be managed and visited together in one single session or view. Also, users may not care about the distribution and source of data at all. The VDM (Virtualize Data Market) connector is aimed at bringing in this feature to openLooKeng.

The VDM connector supports to:

  • Create, update and delete views that combines multiple catalogs
  • Visit real data through the views
  • Manage user privilege through the views
  • Log the use of VDM views by each user

Usage

VDM uses openLooKeng metastore to store its database information. It can be stored either on HDFS or relational database, depending on the implementation of openLooKeng metastore.

Therefore metastore must be configured first.

  • Here is an example of using RDBMS as metastore, create etc/hetu-metastore.properties:
# the type of metastore storage
hetu.metastore.type=jdbc
# jdbc connection address of database
hetu.metastore.db.url=jdbc:mysql://....
# the user name of the database
hetu.metastore.db.user=root
# the password of database 
hetu.metastore.db.password=123456
  • Here is an example of using HDFS as metastore,create etc/hetu-metastore.properties

    Path white list:["/tmp”, “/opt/hetu”, “/opt/openlookeng”, “/etc/hetu”, “/etc/openlookeng”, current workspace]

    Notice:avoid to choose root directory; ../ can’t include in path; if you config node.date_dir, then the current workspace is the parent of node.data_dir; otherwise, the current workspace is the openlookeng server’s directory.

# the type of metastore storage
hetu.metastore.type=hetufilesystem
# profile name of hetu file system
hetu.metastore.hetufilesystem.profile-name=hdfs-config-metastore
# the path of metastore storage in the hetu file system
hetu.metastore.hetufilesystem.path=/etc/openlookeng/metastore

Check filesystem for more information.

For user interface, the connector can be accessed from JDBC or command line interface. Currently VDM only supports schemas and views. Tables are NOT supported.

Schema operations are the same as usual openLooKeng catalogs, including create schema, drop schema, rename schema and show schemas.

Views can be created under a specific schema: create view as ..., drop view.

Example usage:

Configure a data source vdm1 by creating vdm1.properties in etc/catalogs with following contents:

connector.name=vdm

This example creates a schema schema1 in vdm1 catalog, and creates two views from two other different data sources. Note that metastore must be configured in advance (See usage section).

create schema vdm1.schema1;
use vdm1.schema1;
create view view1 as select * from mysql.table.test;
create view view2 as select * from hive.table.test;
select * from view1;

VDM datasource can also be managed through dynamic catalog API. See Dynamic Catalog topic for more information.

All supported CLI queries

Support operationExternal interface (SQL command)
Add VDMcreate catalog(resulful)
Remove VDMdrop catalog(resulful)
Query all VDMshow catalogs
Create schemacreate schema
Delete schemadrop schema
Rename schemarename schema
Query all schemas under VDMshow schemas
Query all views in the schemashow tables
Create/Update Viewcreate [or replace] view
Delete viewdrop view
Query data by viewselect
Query view creation informationshow create view
Query view column informationdescribe
See SQL Statement Syntax topic for more information.

有奖捉虫

“有虫”文档片段

0/500

存在的问题

文档存在风险与错误

● 拼写,格式,无效链接等错误;

● 技术原理、功能、规格等描述和软件不一致,存在错误;

● 原理图、架构图等存在错误;

● 版本号不匹配:文档版本或内容描述和实际软件不一致;

● 对重要数据或系统存在风险的操作,缺少安全提示;

● 排版不美观,影响阅读;

内容描述不清晰

● 描述存在歧义;

● 图形、表格、文字等晦涩难懂;

● 逻辑不清晰,该分类、分项、分步骤的没有给出;

内容获取有困难

● 很难通过搜索引擎,openLooKeng官网,相关博客找到所需内容;

示例代码有错误

● 命令、命令参数等错误;

● 命令无法执行或无法完成对应功能;

内容有缺失

● 关键步骤错误或缺失,无法指导用户完成任务,比如安装、配置、部署等;

● 逻辑不清晰,该分类、分项、分步骤的没有给出

● 图形、表格、文字等晦涩难懂

● 缺少必要的前提条件、注意事项等;

● 描述存在歧义

0/500

您对文档的总体满意度

非常不满意
非常满意

请问是什么原因让您参与到这个问题中

您的邮箱

创Issue赢奖品
根据您的反馈,会自动生成issue模板。您只需点击按钮,创建issue即可。
有奖捉虫