Skip to content

Running on YARN

Jump to bottom Edit New page
Janne Valkealahti edited this page Jan 15, 2015 · 24 revisions

Introduction

The Spring XD distributed runtime (DIRT) supports distribution of processing tasks across multiple nodes. See Running Distributed Mode for information on running Spring XD in distributed mode. One option is to run these nodes on a Hadoop YARN cluster rather than on VMs or physical servers managed by you.

What do you need?

To begin with, you need to have access to a Hadoop cluster running a version based on Apache Hadoop 2.2.0 or later. This includes Apache Hadoop 2.2.0, Pivotal HD 2.0, Hortonworks HDP 2.1 and Cloudera CDH5.

You need a supported transport, see Running Distributed Mode for installation of Redis or Rabbit MQ. Spring XD on YARN currently uses Redis as the default data transport.

You also need Zookeeper running. If your Hadoop cluster doesn’t have Zookeeper installed you need to install and run it specifically for Spring XD. See the Setting up ZooKeeper section of the "Running Distributed Mode" chapter.

Lastly, you need an RDBMs to support batch jobs and JDBC operations.

Download Spring XD on YARN binaries

In addition to the regular spring-xd-<version>-dist.zip files we also distribute a zip file that includes all you need to deploy on YARN. The name of this zip file is spring-xd-<version>-yarn.zip. You can download zip for the current release or download snapshot build. Unzip the downloaded file and you should see a spring-xd-<version>-yarn directory.

Configure your deployment

Configuration options are contained in a config/servers.yml file in the Spring XD YARN install directory. You need to configure the hadoop settings, the transport choice plus redis/rabbit settings, the zookeeper settings and the JDBC datasource properties.

Depending on the distribution used you might need to change the siteYarnAppClasspath and siteMapreduceAppClasspath. We have provided basic settings for the supported distros, you just need to uncomment the ones for the distro you use.

These are the settings used for Hadoop 2.2.0:

spring:
    yarn:
        siteYarnAppClasspath: "$HADOOP_CONF_DIR,$HADOOP_COMMON_HOME/share/hadoop/common/*,$HADOOP_COMMON_HOME/share/hadoop/common/lib/*,$HADOOP_HDFS_HOME/share/hadoop/hdfs/*,$HADOOP_HDFS_HOME/share/hadoop/hdfs/lib/*,$HADOOP_YARN_HOME/share/hadoop/yarn/*,$HADOOP_YARN_HOME/share/hadoop/yarn/lib/*"
        siteMapreduceAppClasspath: "$HADOOP_MAPRED_HOME/share/hadoop/mapreduce/*,$HADOOP_MAPRED_HOME/share/hadoop/mapreduce/lib/*"
XD options

For Spring XD you need to define how many admin servers and containers you need using properties spring.xd.adminServers and spring.xd.containers respectively. You also need to define the HDFS location using property spring.yarn.applicationDir where the Spring XD binary and config files will be stored.

spring:
  xd:
    appmasterMemory: 512M
    adminServers: 1
    adminMemory: 512M
    adminLocality: false
    containers: 3
    containerMemory: 512M
    containerLocality: false
    container:
      groups: yarn
  yarn:
    applicationDir: /xd/app/

More about memory settings in above configuration, see section Configuring YARN memory reservations.

Hadoop settings

You need to specify the host where the YARN Resource Manager is running using spring.hadoop.resourceManagerHost as well as the HDFS URL using spring.hadoop.fsUri.

# Hadoop properties
spring:
  hadoop:
    fsUri: hdfs://localhost:8020
    resourceManagerHost: localhost
    config:
      topology.script.file.name: /path/to/topology-script.sh
Important

Setting hadoop topology.script.file.name property is mandatory if more sophisticated container placement is used to allocate XD admins or containers from a spesific hosts or racks. If this property is not set to match a one used in a hadoop cluster, allocations using hosts and racks will simply fail.
Zookeeper settings

You should specify the Zookeeper connection settings

#Zookeeper properties
#client connect string: host1:port1,host2:port2,...,hostN:portN
zk:
  client:
     connect: localhost:2181
Transport options

You should choose either redis (default) or rabbit as the transport and include the host and port in the properties for the choice you made.

# Transport used
transport: redis
# Redis properties
spring:
  redis:
   port: 6379
   host: localhost
JDBC datasource properties

You should specify the JDBC connection properties based on the RDBMs that you use for the batch jobs and JDBC sink

#Config for use with MySQL - uncomment and edit with relevant values for your environment
spring:
  datasource:
    url: jdbc:mysql://yourDBhost:3306/yourDB
    username: yourUsername
    password: yourPassword
    driverClassName: com.mysql.jdbc.Driver
XD Admin port

On default the property server.port which defines the used port for embedded server is disabled thus falling back to default which is 8080.

#Port that admin-ui is listening on
#server:
#  port: 9393

On YARN it is recommended that you simply set the port to 0 meaning server will automatically choose a random port. This is advisable simply because it will prevent port collission which are usually a little difficult to track down from a cluster. See more instructions from section Connect xd-shell to YARN runtime managed admins how to connect xd-shell to admins managed by YARN.

#Port that admin-ui is listening on
server:
  port: 0

Customizing module configurations

The configurations for all modules can be customized by modifying the file modules.yml in the config directory and then adding it to the modules-config.zip archive in the same directory.

You can run the following command from the config directory to achieve this:

jar -uf modules-config.zip modules.yml

Adding custom modules

There is an empty archive file named custom-modules.zip. You can replace this with your own ZIP archive with the same name and it will be uploaded as part of the deployment to YARN. Place custom module definitions in a modules directory inside this new ZIP archive. Module definitions must follow the Spring XD module semantics.

Modify container logging

Logging configuration for XD admins and containers are defined in files config/xd-admin-logger.properties and config/xd-container-logger.properties respectively. These two files are copied over to hdfs during the deployment. If you want to modify logging configuration either modify source files and do a deployment again or modify files in hdfs directly.

Control XD YARN application lifecycle

Change current directory to be the directory that was unzipped spring-xd-<version>-yarn. To read about runtime configuration and more sophisticated features see section Working with container groups.

Push the Spring XD application binaries and config to HDFS

Run the command

$ bin/xd-yarn push
New version installed
List installed application versions

Run the command

$ bin/xd-yarn pushed
  NAME  PATH
  ----  --------------------
  app   hdfs://node1:8020/xd
Submit the Spring XD YARN application

Run the command

$ bin/xd-yarn submit
New instance submitted with id application_1420911708637_0001
Check the status of YARN apps

You can use the regular yarn command to check the status. Simply run:

$ bin/xd-yarn submitted
  APPLICATION ID                  USER          NAME    QUEUE    TYPE STARTTIME       FINISHTIME  STATE    FINALSTATUS  ORIGINAL TRACKING URL
  ------------------------------  ------------  ------  ------- ----  --------------  ----------  -------  ----------- ---------------------------
  application_1420911708637_0001  jvalkealahti  xd-app  default XD    09/01/15 14:25  N/A         RUNNING  UNDEFINED http://172.16.101.106:49792

You should see one application running named xd-app.

Important

Pay attention to APPLICATION ID listed in output because that is an id used in most of the control commands to communicate to a specific application instance. For example you may have multiple XD YARN runtime instances running.
Kill application

Application can be killed using a kill command.

$ bin/xd-yarn kill -a application_1420905836797_0001
Kill request for application_1420905836797_0001 sent
Using a build-in shell

To get a better and faster command usage a build-in shell can be used to run control commands:

$ bin/xd-yarn shell
Spring YARN Cli (v2.1.0.M3)
Hit TAB to complete. Type 'help' and hit RETURN for help, and 'exit' to quit.
$
clear            clustercreate    clusterdestroy   clusterinfo      clustermodify
clustersinfo     clusterstart     clusterstop      exit             help
kill             prompt           pushed           submit           submitted
$

Connect xd-shell to YARN runtime managed admins

XD admins will register its runtime information into zookeeper and currently this is only place to find out nodes and ports. You can use zookeeper cli to query this information:

[zk: localhost:2181(CONNECTED) 1] ls /xd/admins
[875e2dae-75d9-4b6a-8265-e0eaf0a9cb63]

[zk: localhost:2181(CONNECTED) 0] get
/xd/admins/6f3e6229-9607-4f61-9c54-6788207660c4
{"port":"42898","host":"node6.localdomain","id":"6f3e6229-9607-4f61-9c54-6788207660c4","managementPort":"9393","ip":"172.16.101.106","pid":"10952"}

Then connect xd-shell to this instance:

server-unknown:>admin config server http://node6:42898
Successfully targeted http://node6:42898

xd:>runtime containers
  Container Id                          Host               IP Address      PID    Groups  Custom Attributes
  ------------------------------------  -----------------  --------------  -----  ------  --------------------------------------------------
 6324a9ae-205b-44b9-b851-f0edd7245286  node2.localdomain   172.16.101.102  12284  yarn    {virtualCores=1, memory=512, managementPort=54694}

Configuring YARN memory reservations

YARN Nodemanager is continously tracking how much memory is used by individual YARN containers. If containers are using more memory than what the configuration allows, containers are simply killed by a Nodemanager. Application master controlling the app lifecycle is given a little more freedom meaning that Nodemanager is not that aggressive when making a desicion when a container should be killed.

Lets take a quick look of memory related settings in YARN cluster and in YARN applications. Below xml config is what a default vanilla Apache Hadoop uses for memory related settings. Other distributions may have different defaults.

yarn-site.xml
<configuration>

  <property>
    <name>yarn.nodemanager.pmem-check-enabled</name>
    <value>true</value>
  </property>

  <property>
    <name>yarn.nodemanager.vmem-check-enabled</name>
    <value>true</value>
  </property>

  <property>
    <name>yarn.nodemanager.vmem-pmem-ratio</name>
    <value>2.1</value>
  </property>

  <property>
    <name>yarn.scheduler.minimum-allocation-mb</name>
    <value>1024</value>
  </property>

  <property>
    <name>yarn.scheduler.maximum-allocation-mb</name>
    <value>8192</value>
  </property>

  <property>
    <name>yarn.nodemanager.resource.memory-mb</name>
    <value>8192</value>
  </property>

</configuration>
yarn.nodemanager.pmem-check-enabled

Enables a check for physical memory of a process. This check if enabled is directly tracking amount of memory requested for a YARN container.

yarn.nodemanager.vmem-check-enabled

Enables a check for virtual memory of a process. This setting is one which is usually causing containers of a custom YARN applications to get killed by a node manager. Usually the actual ratio between physical and virtual memory is higher than a default 2.1 or bugs in a OS is causing wrong calculation of a used virtual memory.

yarn.nodemanager.vmem-pmem-ratio

Defines a ratio of allowed virtual memory compared to physical memory. This ratio simply defines how much virtual memory a process can use but the actual tracked size is always calculated from a physical memory limit.

yarn.scheduler.minimum-allocation-mb

Defines a minimum allocated memory for container.

Important

This setting also indirectly defines what is the actual physical memory limit requested during a container allocation. Actual physical memory limit is always going to be multiple of this setting rounded to upper bound. For example if this setting is left to default 1024 and container is requested with 512M, 1024M is going to be used. However if requested size is 1100M, actual size is set to 2048M.
yarn.scheduler.maximum-allocation-mb

Defines a maximum allocated memory for container.

yarn.nodemanager.resource.memory-mb

Defines how much memory a node controlled by a node manager is allowed to allocate. This setting should be set to amount of which OS is able give to YARN managed processes in a way which doesn’t cause OS to swap, etc.

Tip

If testing XD YARN runtime on a single computer with a multiple VM based hadoop cluster a pro tip is to set both yarn.nodemanager.pmem-check-enabled and yarn.nodemanager.vmem-check-enabled to false, set yarn.scheduler.minimum-allocation-mb much lower to either 256 or 512 and yarn.nodemanager.resource.memory-mb 15%-20% below a defined VM memory.

We have three memory settings for components participating XD YARN runtime. You can use configuration properties spring.xd.appmasterMemory, spring.xd.adminMemory and spring.xd.containerMemory respectively.

spring:
  xd:
    appmasterMemory: 512M
    adminMemory: 512M
    containerMemory: 512M

Working with container groups

Container grouping and clustering is more sophisticated feature which allows better control of XD admins and containers at runtime. Basic features are:

  • Control members in a groups.

  • Control lifecycle state for group as whole.

  • Create groups dynamically.

  • Re-start failed containers.

XD YARN Runtime has a few built-in groups to get you started. There are two groups admin and container created by default which both are lauching exactly one container chosen randomly from YARN cluster.

List existing groups

Run the command:

$ bin/xd-yarn clustersinfo -a application_1420911708637_0001
  CLUSTER ID
  ----------
  container
  admin
Get status of a group

Run the command:

bin/xd-yarn clusterinfo -a application_1420911708637_0001 -c admin
  CLUSTER STATE  MEMBER COUNT
  -------------  ------------
  RUNNING        1

Or to get verbose output:

$ bin/xd-yarn clusterinfo -a application_1420911708637_0001 -c admin -v
  CLUSTER STATE  MEMBER COUNT  ANY PROJECTION  HOSTS PROJECTION  RACKS PROJECTION  ANY SATISFY  HOSTS SATISFY  RACKS SATISFY
  -------------  ------------  --------------  ----------------  ----------------  -----------  -------------  -------------
  RUNNING        1             1               {}                {}                0            {}             {}
Control group state

Run the commands to stop group, list its status, start group and finally list status again:

$ bin/xd-yarn clusterinfo -a application_1420911708637_0001 -c container
  CLUSTER STATE  MEMBER COUNT
  -------------  ------------
  RUNNING        1

$ bin/xd-yarn clusterstop -a application_1420911708637_0001 -c container
Cluster container stopped.

$ bin/xd-yarn clusterinfo -a application_1420911708637_0001 -c container
  CLUSTER STATE  MEMBER COUNT
  -------------  ------------
  STOPPED        0

$ bin/xd-yarn clusterstart -a application_1420911708637_0001 -c container
Cluster container started.

$ bin/xd-yarn clusterinfo -a application_1420911708637_0001 -c container
  CLUSTER STATE  MEMBER COUNT
  -------------  ------------
  RUNNING        1
Modify group configuration

In these commans we first ramp up container count and then ramp it down:

18:19 $ bin/xd-yarn clusterinfo -a application_1420911708637_0001 -c container
  CLUSTER STATE  MEMBER COUNT
  -------------  ------------
  RUNNING        1

$ bin/xd-yarn clustermodify -a application_1420911708637_0001 -c container -w 3
Cluster container modified.

$ bin/xd-yarn clusterinfo -a application_1420911708637_0001 -c container
  CLUSTER STATE  MEMBER COUNT
  -------------  ------------
  RUNNING        3

$ bin/xd-yarn clustermodify -a application_1420911708637_0001 -c container -w 2
Cluster container modified.

$ bin/xd-yarn clusterinfo -a application_1420911708637_0001 -c container
  CLUSTER STATE  MEMBER COUNT
  -------------  ------------
  RUNNING        2
Note

In above example we used option -w which is a shortcut for defining YARN allocation which uses a wildcard requests allowing containers to be requested from any host.
Create a new group

When you want to create a new group that is because you need to add new XD admin or container nodes to a current system with a different settings. These setting usually differ by a colocation of containers. More about built-in group configuration refer to section Built-in group configurations.

Run the command:

$ bin/xd-yarn clustercreate -a application_1420911708637_0001 -c custom -i container-nolocality-template -p default -w 2
Cluster custom created.

$ bin/xd-yarn clusterinfo -a application_1420911708637_0001 -c custom
  CLUSTER STATE  MEMBER COUNT
  -------------  ------------
  INITIAL        0

$ bin/xd-yarn clusterstart -a application_1420911708637_0001 -c custom
Cluster custom started.

$ bin/xd-yarn clusterinfo -a application_1420911708637_0001 -c custom
  CLUSTER STATE  MEMBER COUNT
  -------------  ------------
  RUNNING        2

To create group with two containers on node5 and one on node6 run command:

$ bin/xd-yarn clustercreate -a application_1420911708637_0001 -c custom -i container-locality-template -p default -y "{hosts:{node6: 1,node5: 2}}"
Cluster custom created.

$ bin/xd-yarn -a application_1420911708637_0001 -c custom -v
  CLUSTER STATE  MEMBER COUNT  ANY PROJECTION  HOSTS PROJECTION    RACKS PROJECTION  ANY SATISFY  HOSTS SATISFY       RACKS SATISFY
  -------------  ------------  --------------  ------------------  ----------------  -----------  ------------------  -------------
  INITIAL        0             0               {node5=2, node6=1}  {}                0            {node5=2, node6=1}  {}
Destroy a group

Run the commands:

$ bin/xd-yarn clustersinfo -a application_1420911708637_0001
  CLUSTER ID
  ----------
  container
  admin

$ bin/xd-yarn clusterinfo -a application_1420911708637_0001 -c container
  CLUSTER STATE  MEMBER COUNT
  -------------  ------------
  RUNNING        1

$ bin/xd-yarn clusterstop -a application_1420911708637_0001 -c container
Cluster container stopped.

$ bin/xd-yarn clusterinfo -a application_1420911708637_0001 -c container
  CLUSTER STATE  MEMBER COUNT
  -------------  ------------
  STOPPED        0

$ bin/xd-yarn clusterdestroy -a application_1420911708637_0001 -c container
Cluster container destroyed.

$ bin/xd-yarn clustersinfo -a application_1420911708637_0001
  CLUSTER ID
  ----------
  admin
Note

Group can only destroyed if its status is STOPPED or INITIAL.
Built-in group configurations

Few groups are already defined where admin and container are enabled automatically. Other groups are disabled and thus working as a blueprints which can be used to create groups manually.

admin

Default group definition for XD admins.

container

Default group definition for XD containers.

admin-nolocality-template

Blueprint with relax localization. Use this to create a groups if you plan to use any matching.

admin-locality-template

Blueprint with no relax localization. Use this to create a groups if you plan to use hosts or racks matching.

container-nolocality-template

Blueprint with relax localization. Use this to create a groups if you plan to use any matching.

container-locality-template

Blueprint with no relax localization. Use this to create a groups if you plan to use hosts or racks matching.

Configuration examples

This section contains examples of usual use cases for custom configurations.

Run containers on a specific hosts

Below configuration sets default XD container to exist on node1 and node2.

xd:
  containerLocality: true
spring:
  yarn:
    appmaster:
      containercluster:
        clusters:
          container:
            projection:
              data: {any: 0, hosts: {node1: 1, node2: 1}}
Run admins on a specific racks

Below configuration sets default XD admins to exist on /rack1 and /rack2.

xd:
  adminLocality: true
spring:
  yarn:
    appmaster:
      containercluster:
        clusters:
          admin:
            projection:
              data: {any: 0, racks: {/rack1: 1, /rack2: 1}}
Disable default admin and container groups

Existing built-in groups admin and container can be disabled by setting their projection types to null.

spring:
  yarn:
    appmaster:
      containercluster:
        clusters:
          admin:
            projection:
              type: null
          container:
            projection:
              type: null

xd-yarn command synopsis

push
xd-yarn push - Push new application version

usage: xd-yarn push [options]

Option                     Description
------                     -----------
-v, --application-version  Application version (default: app)
pushed
xd-yarn pushed - List pushed applications

usage: xd-yarn pushed [options]

No options specified
submit
xd-yarn submit - Submit application

usage: xd-yarn submit [options]

Option                     Description
------                     -----------
-v, --application-version  Application version (default: app)
submitted
xd-yarn submitted - List submitted applications

usage: xd-yarn submitted [options]

Option                   Description
------                   -----------
-t, --application-type   Application type (default: XD)
-v, --verbose [Boolean]  Verbose output (default: true)
kill
xd-yarn kill - Kill application

usage: xd-yarn kill [options]

Option                Description
------                -----------
-a, --application-id  Specify YARN application id
clustersinfo
xd-yarn clustersinfo - List clusters

usage: xd-yarn clustersinfo [options]

Option                Description
------                -----------
-a, --application-id  Specify YARN application id
clusterinfo
xd-yarn clusterinfo - List cluster info

usage: xd-yarn clusterinfo [options]

Option                   Description
------                   -----------
-a, --application-id     Specify YARN application id
-c, --cluster-id         Specify cluster id
-v, --verbose [Boolean]  Verbose output (default: true)
clustercreate
xd-yarn clustercreate - Create cluster

usage: xd-yarn clustercreate [options]

Option                  Description
------                  -----------
-a, --application-id    Specify YARN application id
-c, --cluster-id        Specify cluster id
-g, --container-groups  Container groups
-h, --projection-hosts  Projection hosts counts
-i, --cluster-def       Specify cluster def id
-p, --projection-type   Projection type
-r, --projection-racks  Projection racks counts
-w, --projection-any    Projection any count
-y, --projection-data   Raw projection data
clusterdestroy
xd-yarn clusterdestroy - Destroy cluster

usage: xd-yarn clusterdestroy [options]

Option                Description
------                -----------
-a, --application-id  Specify YARN application id
-c, --cluster-id      Specify cluster id
clustermodify
xd-yarn clustermodify - Modify cluster

usage: xd-yarn clustermodify [options]

Option                  Description
------                  -----------
-a, --application-id    Specify YARN application id
-c, --cluster-id        Specify cluster id
-h, --projection-hosts  Projection hosts counts
-r, --projection-racks  Projection racks counts
-w, --projection-any    Projection any count
-y, --projection-data   Raw projection data
clusterstart
xd-yarn clusterstart - Start cluster

usage: xd-yarn clusterstart [options]

Option                Description
------                -----------
-a, --application-id  Specify YARN application id
-c, --cluster-id      Specify cluster id
clusterstop
xd-yarn clusterstop - Stop cluster

usage: xd-yarn clusterstop [options]

Option                Description
------                -----------
-a, --application-id  Specify YARN application id
-c, --cluster-id      Specify cluster id

Introduction to YARN resource allocation

This section describes some background of how YARN resource allocation works, what are the limitations of it and more importantly how it reflects into XD YARN runtime.

Note

More detailed info of resource allocation can be found from a Spring for Apache Hadoop reference documentation.

YARN as having a strong roots from original MapReduce framework is imposing relatively strange concepts of where containers are about to be executed. In a MapReduce world every map and reduce tasks are executed in its own container where colocation is usually determined by a physical location of a HDFS file block map or reduce tasks are accessing. This is introducing a concepts of allocating containers on any hosts, specific hosts or specific racks. Usually YARN is trying to place container as close as possible to a physical location to minimize network IO so i.e. if host cannot be chosen, rack is chosen instead assuming a whole rack is connected together with a fast switch.

For custom YARN applications like XD YARN runtime this doesn’t necessarily make that much sense because we’re not hard-tied to HDFS file blocks. What makes sense is that we can still place containers on different racks to get better high availability in case whole rack goes down or if specific containers needs to exist on specific hosts to access either custom physical or network resources. Good example of having a need to execute something on a specific host is either a disk access or outbound internet access if cluster is highly secured.

One other YARN resource allocation concept worth mentioning is relaxation of container locality. This simply means that if resources are requested from hosts or racks, YARN will relax those requests if resources cannot be allocated immediately. Turning relax flag off guarantees that containers will be allocated from hosts or racks. Though these requests will then wait forever if allocation cannot be done.

Contents

Developing Modules and Extensions

Configuration Guidelines

Administration

Appendices

Clone this wiki locally