openLooKeng on YARN

The openLooKeng-on-YARN deployment package provides automation scripts and configuration templates to deploy openLooKeng clusters on YARN.

Features

1. Launch multiple openLooKeng nodes on a single machine
2. Deploy multiple clusters
3. Scale up/down openLooKeng workers in a cluster
4. Query openLooKeng cluster status
5. Tear-down openLooKeng cluster
6. High availability deployment with multiple coordinators, using Nginx as a load balancer

Pre-requisites

1. A Hadoop environment with YARN service framework.
   1. Apache Hadoop (versions tested: 3.1.4, 3.3.1)
   2. User access right to HDFS
2. Environment Requirements
   1. Python3, version 3.6 or higher
      1. hdfs 2.6.0
      2. requests 2.22.0
      3. hdfs[kerberos] 2.6.0
      4. requests_kerberos 0.12.0

openLooKeng cluster on YARN

The ./bin/olk_on_yarn.py script provides one-stop deployment management for openLooKeng cluster on YARN

Administrator Configuration

The following configuration files would be edited by the administrator to set the correct environment files and also the cluster configurations.

HDFS Filesystem - Mandatory

Correct configurations will be required for openLooKeng instances to access the HDFS data. Administrator will be required to provide the correct configurations for the following files. Under the conf/etc_template/filesystem/ directory

1. core-site.xml
2. hdfs-site.xml

Deployment

Automation scripts can be configured to provide parameters such as

1. openLooKeng Cluster name and service port
2. Number of workers and/ or coordinators
3. High Availability enable/ disable
4. YARN Resource Manager socket address

Example configuration for automation scripts

• olk_on_yarn_default.json (default)

Provide configuration files for the openLooKeng coordinator under the following directory:

• conf/etc_template/coordinator

Provide configuration files for the openLooKeng worker under the directory

• conf/etc_template/worker

Plugins & Catalogs

Additional configuration files for openLooKeng should be provided in the coordinator and worker directories.

Derivative work

The original copy of bin/launcher.py can be obtained from the airlift/airlift project https://github.com/airlift/airlift/blob/master/launcher/src/main/scripts/bin/launcher.py