Connectors

Connectors are the source of all data for queries in openLooKeng. Even if your data source doesn't have underlying tables backing it, as long as you adapt your data source to the API expected by openLooKeng, you can write queries against this data.

ConnectorFactory

Instances of your connector are created by a ConnectorFactory instance which is created when openLooKeng calls getConnectorFactory() on the plugin. The connector factory is a simple interface responsible for creating an instance of a Connector object that returns instances of the following services:

  • ConnectorMetadata
  • ConnectorSplitManager
  • ConnectorHandleResolver
  • ConnectorRecordSetProvider

ConnectorMetadata

The connector metadata interface has a large number of important methods that are responsible for allowing openLooKeng to look at lists of schemas, lists of tables, lists of columns, and other metadata about a particular data source.

This interface is too big to list in this documentation, but if you are interested in seeing strategies for implementing these methods, look at the example-http and the Cassandra connector. If your underlying data source supports schemas, tables and columns, this interface should be straightforward to implement. If you are attempting to adapt something that is not a relational database (as the Example HTTP connector does), you may need to get creative about how you map your data source to openLooKeng's schema, table, and column concepts.

ConnectorSplitManager

The split manager partitions the data for a table into the individual chunks that openLooKeng will distribute to workers for processing. For example, the Hive connector lists the files for each Hive partition and creates one or more split per file. For data sources that don't have partitioned data, a good strategy here is to simply return a single split for the entire table. This is the strategy employed by the Example HTTP connector.

ConnectorRecordSetProvider

Given a split and a list of columns, the record set provider is responsible for delivering data to the openLooKeng execution engine. It creates a RecordSet, which in turn creates a RecordCursor that is used by openLooKeng to read the column values for each row.

有奖捉虫

“有虫”文档片段

0/500

存在的问题

文档存在风险与错误

● 拼写,格式,无效链接等错误;

● 技术原理、功能、规格等描述和软件不一致,存在错误;

● 原理图、架构图等存在错误;

● 版本号不匹配:文档版本或内容描述和实际软件不一致;

● 对重要数据或系统存在风险的操作,缺少安全提示;

● 排版不美观,影响阅读;

内容描述不清晰

● 描述存在歧义;

● 图形、表格、文字等晦涩难懂;

● 逻辑不清晰,该分类、分项、分步骤的没有给出;

内容获取有困难

● 很难通过搜索引擎,openLooKeng官网,相关博客找到所需内容;

示例代码有错误

● 命令、命令参数等错误;

● 命令无法执行或无法完成对应功能;

内容有缺失

● 关键步骤错误或缺失,无法指导用户完成任务,比如安装、配置、部署等;

● 逻辑不清晰,该分类、分项、分步骤的没有给出

● 图形、表格、文字等晦涩难懂

● 缺少必要的前提条件、注意事项等;

● 描述存在歧义

0/500

您对文档的总体满意度

非常不满意
非常满意

请问是什么原因让您参与到这个问题中

您的邮箱

创Issue赢奖品
根据您的反馈,会自动生成issue模板。您只需点击按钮,创建issue即可。
有奖捉虫