org.apache.slider.server.appmaster

Class SliderAppMaster

java.lang.Object

org.apache.hadoop.service.AbstractService

org.apache.hadoop.service.CompositeService

org.apache.slider.server.services.workflow.WorkflowCompositeService

org.apache.slider.server.services.utility.LaunchedWorkflowCompositeService

org.apache.slider.server.services.utility.AbstractSliderLaunchedService

org.apache.slider.server.appmaster.SliderAppMaster

All Implemented Interfaces:

Closeable, AutoCloseable, org.apache.hadoop.service.Service, org.apache.hadoop.service.ServiceStateChangeListener, org.apache.hadoop.yarn.client.api.async.AMRMClientAsync.CallbackHandler, org.apache.hadoop.yarn.client.api.async.NMClientAsync.CallbackHandler, RoleKeys, SliderExitCodes, SliderKeys, SliderXmlConfKeys, LauncherExitCodes, RunService, ProviderCompleted, AppMasterActionOperations, RMOperationHandlerActions, ServiceParent

public class SliderAppMaster
extends AbstractSliderLaunchedService
implements org.apache.hadoop.yarn.client.api.async.AMRMClientAsync.CallbackHandler, org.apache.hadoop.yarn.client.api.async.NMClientAsync.CallbackHandler, RunService, SliderExitCodes, SliderKeys, org.apache.hadoop.service.ServiceStateChangeListener, RoleKeys, ProviderCompleted, AppMasterActionOperations

This is the AM, which directly implements the callbacks from the AM and NM

Nested Class Summary

Nested classes/interfaces inherited from class org.apache.hadoop.service.CompositeService

org.apache.hadoop.service.CompositeService.CompositeServiceShutdownHook

Nested classes/interfaces inherited from interface org.apache.hadoop.service.Service

org.apache.hadoop.service.Service.STATE

Field Summary

Fields

Modifier and Type	Field and Description
protected Map<org.apache.hadoop.yarn.api.records.ApplicationAccessType,String>	applicationACLs App ACLs
static String	E_TRIGGERED_LAUNCH_FAILURE Error string on chaos monkey launch failure action: "Chaos monkey triggered launch failure"
static int	HEARTBEAT_INTERVAL
protected static org.slf4j.Logger	log
protected static org.slf4j.Logger	LOG_YARN log for YARN events
com.codahale.metrics.MetricRegistry	metrics metrics registry
org.apache.hadoop.yarn.client.api.async.NMClientAsync	nmClientAsync Handle to communicate with the Node Manager
static int	NUM_RPC_HANDLERS
static String	SERVICE_CLASSNAME
static String	SERVICE_CLASSNAME_SHORT

Fields inherited from class org.apache.hadoop.service.CompositeService

STOP_ONLY_STARTED_SERVICES

Fields inherited from interface org.apache.slider.common.SliderExitCodes

_EXIT_CODE_BASE, EXIT_APPLICATION_IN_USE, EXIT_BAD_CONFIGURATION, EXIT_BAD_STATE, EXIT_DEPLOYMENT_FAILED, EXIT_INSTANCE_EXISTS, EXIT_PROCESS_FAILED, EXIT_TIMED_OUT, EXIT_UNKNOWN_INSTANCE, EXIT_YARN_SERVICE_FAILED, EXIT_YARN_SERVICE_FINISHED_WITH_ERROR, EXIT_YARN_SERVICE_KILLED

Fields inherited from interface org.apache.slider.core.main.LauncherExitCodes

EXIT_CLIENT_INITIATED_SHUTDOWN, EXIT_COMMAND_ARGUMENT_ERROR, EXIT_CONFLICT, EXIT_CONNECTIVITY_PROBLEM, EXIT_EXCEPTION_THROWN, EXIT_FALSE, EXIT_FORBIDDEN, EXIT_FOUND, EXIT_INTERNAL_ERROR, EXIT_INTERRUPTED, EXIT_MOVED, EXIT_NOT_ACCEPTABLE, EXIT_NOT_FOUND, EXIT_NOT_MODIFIED, EXIT_OPERATION_NOT_ALLOWED, EXIT_OTHER_FAILURE, EXIT_SERVICE_UNAVAILABLE, EXIT_SUCCESS, EXIT_TASK_LAUNCH_FAILURE, EXIT_UNAUTHORIZE, EXIT_UNIMPLEMENTED, EXIT_UNSUPPORTED_VERSION, EXIT_USAGE

Fields inherited from interface org.apache.slider.common.SliderKeys

ADDONS_DIR, AGENT_TAR, AM_DIR_PREFIX, AM_FILTER_NAME, APP_CONTAINER_HEARTBEAT_INTERVAL_SEC, APP_CONTAINER_RELEASE_TIMEOUT, APP_DEF_DIR, APP_TYPE, APP_VERSION, APP_VERSION_UNKNOWN, CLUSTER_DIRECTORY, COMP_KEYSTORE_PASSWORD_ALIAS_DEFAULT, COMP_KEYSTORE_PASSWORD_ALIAS_KEY, COMP_KEYSTORE_PASSWORD_PROPERTY_KEY, COMP_STORES_REQUIRED_KEY, COMP_TRUSTSTORE_PASSWORD_ALIAS_DEFAULT, COMP_TRUSTSTORE_PASSWORD_ALIAS_KEY, COMP_TRUSTSTORE_PASSWORD_PROPERTY_KEY, COMPONENT_AM, CRT_FILE_NAME, CRT_PASS_FILE_NAME, CSR_FILE_NAME, DATA_DIR_NAME, DEFAULT_APP_PKG, DEFAULT_GC_OPTS, DEFAULT_JVM_HEAP, DEFAULT_YARN_MEMORY, GENERATED_CONF_DIR_NAME, GSON_JAR, HADOOP_PROXY_USER, HADOOP_USER_NAME, HDP_VERSION_PROP_NAME, HISTORY_DIR_NAME, HISTORY_FILENAME_CREATION_PATTERN, HISTORY_FILENAME_GLOB_PATTERN, HISTORY_FILENAME_MATCH_PATTERN, HISTORY_FILENAME_PREFIX, HISTORY_FILENAME_SUFFIX, INFRA_DIR_NAME, JCOMMANDER_JAR, JVM_ENABLE_ASSERTIONS, JVM_ENABLE_SYSTEM_ASSERTIONS, JVM_FORCE_IPV4, JVM_JAVA_HEADLESS, KEY_ALLOWED_PORT_RANGE, KEY_AM_ALLOWED_PORT_RANGE, KEY_FILE_NAME, KEYSTORE_FILE_NAME, KEYTAB_DIR, LOCAL_TARBALL_INSTALL_SUBDIR, LOG4J_PROP_FILENAME, LOG4J_SERVER_PROP_FILENAME, PACKAGE_DIRECTORY, PASS_LEN, PROPAGATE_RESOURCE_OPTION, PROPAGATED_CONF_DIR_NAME, PROPERTY_CONF_DIR, PROPERTY_LIB_DIR, PYTHONPATH, ROLE_AM_PRIORITY_INDEX, SECURITY_DIR, SLIDER_BASE_DIRECTORY, SLIDER_CLASSPATH_EXTRA, SLIDER_CLIENT_XML, SLIDER_DEPENDENCY_DIR, SLIDER_DEPENDENCY_DIR_PERMISSIONS, SLIDER_DEPENDENCY_HDP_PARENT_DIR, SLIDER_DEPENDENCY_LOCALIZED_DIR_LINK, SLIDER_DEPENDENCY_TAR_GZ_FILE_EXT, SLIDER_DEPENDENCY_TAR_GZ_FILE_NAME, SLIDER_JAR, SLIDER_JVM_OPTS, SLIDER_PASSPHRASE, SLIDER_SERVER_XML, SLIDER_XML, SNAPSHOT_CONF_DIR_NAME, STDERR_AM, STDOUT_AM, SUBMITTED_CONF_DIR, SYSPROP_LOG_DIR, SYSPROP_LOG4J_CONFIGURATION, TMP_DIR_PREFIX, TMP_LOGDIR_PREFIX, YARN_CONTAINER_PATH

Fields inherited from interface org.apache.slider.common.SliderXmlConfKeys

CLUSTER_DIRECTORY_PERMISSIONS, DATA_DIRECTORY_PERMISSIONS, DEFAULT_CLUSTER_DIRECTORY_PERMISSIONS, DEFAULT_DATA_DIRECTORY_PERMISSIONS, DEFAULT_YARN_QUEUE, DEFAULT_YARN_QUEUE_PRIORITY, DFS_DATANODE_KERBEROS_PRINCIPAL_KEY, DFS_NAMENODE_DELEGATION_KEY_UPDATE_INTERVAL_DEFAULT, DFS_NAMENODE_DELEGATION_KEY_UPDATE_INTERVAL_KEY, DFS_NAMENODE_DELEGATION_TOKEN_ALWAYS_USE_DEFAULT, DFS_NAMENODE_DELEGATION_TOKEN_ALWAYS_USE_KEY, DFS_NAMENODE_DELEGATION_TOKEN_MAX_LIFETIME_DEFAULT, DFS_NAMENODE_DELEGATION_TOKEN_MAX_LIFETIME_KEY, DFS_NAMENODE_DELEGATION_TOKEN_RENEW_INTERVAL_DEFAULT, DFS_NAMENODE_DELEGATION_TOKEN_RENEW_INTERVAL_KEY, DFS_NAMENODE_DU_RESERVED_KEY, DFS_NAMENODE_KERBEROS_PRINCIPAL_KEY, DFS_NAMENODE_KEYTAB_FILE_KEY, FS_DEFAULT_NAME_CLASSIC, HADOOP_HTTP_FILTER_INITIALIZERS, IPC_CLIENT_FALLBACK_TO_SIMPLE_AUTH, IPC_CLIENT_RETRY_POLICY_ENABLED_DEFAULT, IPC_CLIENT_RETRY_POLICY_SPEC_DEFAULT, KEY_AM_KEYTAB_LOCAL_PATH, KEY_AM_LAUNCH_ENV, KEY_AM_LOGIN_KEYTAB_NAME, KEY_AM_LOGIN_KEYTAB_REQUIRED, KEY_AM_RESTART_LIMIT, KEY_HDFS_KEYTAB_DIR, KEY_IPC_CLIENT_RETRY_POLICY_ENABLED, KEY_IPC_CLIENT_RETRY_POLICY_SPEC, KEY_KERBEROS_PRINCIPAL, KEY_KEYSTORE_LOCATION, KEY_KEYTAB_PRINCIPAL, KEY_PROTOCOL_ACL, KEY_PROVIDER, KEY_SECURITY_ENABLED, KEY_SLIDER_AM_DEPENDENCY_CHECKS_DISABLED, KEY_SLIDER_BASE_PATH, KEY_TEMPLATE_ORIGIN, KEY_YARN_QUEUE, KEY_YARN_QUEUE_PRIORITY, PREFIX_PROVIDER, PYTHON_EXECUTABLE_PATH, REGISTRY_PATH, REGISTRY_ZK_QUORUM, X_DEV_INSECURE_DEFAULT, X_DEV_INSECURE_REQUIRED, X_DEV_INSECURE_WS

Fields inherited from interface org.apache.slider.api.RoleKeys

ENV_PREFIX, GC_OPTS, JVM_HEAP, JVM_OPTS, ROLE_ACTUAL_INSTANCES, ROLE_ADDITIONAL_ARGS, ROLE_FAILED_INSTANCES, ROLE_FAILED_RECENTLY_INSTANCES, ROLE_FAILED_STARTING_INSTANCES, ROLE_NAME, ROLE_NODE_FAILED_INSTANCES, ROLE_PENDING_AA_INSTANCES, ROLE_PREEMPTED_INSTANCES, ROLE_RELEASING_INSTANCES, ROLE_REQUESTED_INSTANCES

Constructor Summary

Constructors

Constructor and Description
SliderAppMaster() Service Constructor

Method Summary

Methods

Modifier and Type	Method and Description
void	addContainerRequest(org.apache.hadoop.yarn.client.api.AMRMClient.ContainerRequest req) Issue a container request
org.apache.hadoop.conf.Configuration	bindArgs(org.apache.hadoop.conf.Configuration config, String... args) pick up the args from the service launcher
int	cancelContainerRequests(org.apache.hadoop.yarn.api.records.Priority priority1, org.apache.hadoop.yarn.api.records.Priority priority2, int count) Remove a container request
void	cancelSingleRequest(org.apache.hadoop.yarn.client.api.AMRMClient.ContainerRequest request) Cancel a specific request
void	escalateOutstandingRequests() Escalate operation as triggered by external timer.
void	eventCallbackEvent(Object parameter)
void	execute(List<AbstractRMOperation> operations) Execute an entire list of operations
void	executeNodeReview(String reason) Look at where the current node state is -and whether it should be changed
void	flexCluster(ConfTree resources) Implementation of cluster flexing.
AppState	getAppState() Get the application state
ClusterDescription	getClusterDescription() This is the status, the live model
SliderFileSystem	getClusterFS() Get the filesystem of this cluster
String	getGeneratedConfDir() Get the path to the DFS configuration that is defined in the cluster specification
MapOperations	getGlobalInternalOptions() Get the global internal options for the AM
AggregateConf	getInstanceDefinition()
File	getLocalConfDir() Build the configuration directory passed in or of the target FS
static org.slf4j.Logger	getLog() Get the AM log
float	getProgress() heartbeat operation; return the ratio of requested to actual
ProviderService	getProviderService()
Object	getProxy(Class protocol, InetSocketAddress addr)
void	handleReviewAndFlexApplicationSize(ReviewAndFlexApplicationSize action) Handle the event requesting a review ...
org.apache.hadoop.service.Service	initAndAddService(org.apache.hadoop.service.Service service) Initialize a newly created service then add it.
protected void	launchProviderService(AggregateConf instanceDefinition, File confDir) Launch the provider service
protected void	login(String principal, File localKeytabFile)
static void	main(String[] args) This is the main entry point for the service launcher.
void	onAMStop(ActionStopSlider stopActionRequest) Signal that the AM is complete
void	onContainersAllocated(List<org.apache.hadoop.yarn.api.records.Container> allocatedContainers) Callback event when a container is allocated.
void	onContainersCompleted(List<org.apache.hadoop.yarn.api.records.ContainerStatus> completedContainers)
void	onContainerStarted(org.apache.hadoop.yarn.api.records.ContainerId containerId, Map<String,ByteBuffer> allServiceResponse)
void	onContainerStatusReceived(org.apache.hadoop.yarn.api.records.ContainerId containerId, org.apache.hadoop.yarn.api.records.ContainerStatus containerStatus)
void	onContainerStopped(org.apache.hadoop.yarn.api.records.ContainerId containerId)
void	onError(Throwable e)
void	onExceptionInThread(Thread thread, Throwable exception) Handle any exception in a thread.
void	onGetContainerStatusError(org.apache.hadoop.yarn.api.records.ContainerId containerId, Throwable t)
void	onNodesUpdated(List<org.apache.hadoop.yarn.api.records.NodeReport> updatedNodes) Monitored nodes have been changed
void	onShutdownRequest() RM wants to shut down the AM
void	onStartContainerError(org.apache.hadoop.yarn.api.records.ContainerId containerId, Throwable t)
void	onStopContainerError(org.apache.hadoop.yarn.api.records.ContainerId containerId, Throwable t)
void	onUpgradeContainers(ActionUpgradeContainers upgradeContainersRequest) Signal that containers are being upgraded.
void	providerLostContainer(org.apache.hadoop.yarn.api.records.ContainerId containerId) report container loss.
void	queue(AsyncAction action) Queue an action for immediate execution in the executor thread
boolean	registerComponent(org.apache.hadoop.yarn.api.records.ContainerId id, String description) Handler for action Register/re-register an ephemeral container that is already in the app state
void	registerServiceInstance(String instanceName, org.apache.hadoop.yarn.api.records.ApplicationId appId) This registers the service instance and its external values
void	releaseAssignedContainer(org.apache.hadoop.yarn.api.records.ContainerId containerId) Release an assigned container
int	runService() this is called by service launcher; when it returns the application finishes
void	schedule(AsyncAction action) Schedule an action
void	serviceInit(org.apache.hadoop.conf.Configuration conf)
protected void	serviceStart()
protected void	setupInitialRegistryPaths() TODO: purge this once RM is doing the work
void	signalAMComplete(ActionStopSlider stopActionRequest) Signal that the AM is complete ..
void	startContainer(org.apache.hadoop.yarn.api.records.Container container, org.apache.hadoop.yarn.api.records.ContainerLaunchContext ctx, RoleInstance instance) Async start container request
void	stateChanged(org.apache.hadoop.service.Service service) Received on listening service termination.
protected Integer	stopForkedProcess() stop forked process if it the running process var is not null
void	unregisterComponent(org.apache.hadoop.yarn.api.records.ContainerId id) Handler for UnregisterComponentInstance unregister a component.
protected void	validateLoginUser(org.apache.hadoop.security.UserGroupInformation user) Ensure that the user is generated from a keytab and has no HDFS delegation tokens.

Methods inherited from class org.apache.slider.server.services.utility.AbstractSliderLaunchedService

createRegistryOperationsInstance, lookupZKQuorum, require, requireArgumentSet, startRegistryOperationsService

Methods inherited from class org.apache.slider.server.services.utility.LaunchedWorkflowCompositeService

addService, deployChildService, getArgv, isSet, isUnset

Methods inherited from class org.apache.hadoop.service.CompositeService

addIfService, getServices, removeService, serviceStop

Methods inherited from class org.apache.hadoop.service.AbstractService

close, getBlockers, getConfig, getFailureCause, getFailureState, getLifecycleHistory, getName, getServiceState, getStartTime, init, isInState, noteFailure, putBlocker, registerGlobalListener, registerServiceListener, removeBlocker, setConfig, start, stop, toString, unregisterGlobalListener, unregisterServiceListener, waitForServiceToStop

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

Methods inherited from interface org.apache.hadoop.service.Service

close, getBlockers, getConfig, getFailureCause, getFailureState, getLifecycleHistory, getName, getServiceState, getStartTime, init, isInState, registerServiceListener, start, stop, unregisterServiceListener, waitForServiceToStop

Methods inherited from interface org.apache.slider.server.services.workflow.ServiceParent

getServices

Field Detail

log

protected static final org.slf4j.Logger log

LOG_YARN

protected static final org.slf4j.Logger LOG_YARN

log for YARN events

SERVICE_CLASSNAME_SHORT

public static final String SERVICE_CLASSNAME_SHORT

See Also:

Constant Field Values

SERVICE_CLASSNAME

public static final String SERVICE_CLASSNAME

See Also:

Constant Field Values

HEARTBEAT_INTERVAL

public static final int HEARTBEAT_INTERVAL

See Also:

Constant Field Values

NUM_RPC_HANDLERS

public static final int NUM_RPC_HANDLERS

See Also:

Constant Field Values

metrics

public com.codahale.metrics.MetricRegistry metrics

metrics registry

E_TRIGGERED_LAUNCH_FAILURE

public static final String E_TRIGGERED_LAUNCH_FAILURE

Error string on chaos monkey launch failure action: "Chaos monkey triggered launch failure"

See Also:

Constant Field Values

nmClientAsync

public org.apache.hadoop.yarn.client.api.async.NMClientAsync nmClientAsync

Handle to communicate with the Node Manager

applicationACLs

protected Map<org.apache.hadoop.yarn.api.records.ApplicationAccessType,String> applicationACLs

App ACLs

Constructor Detail

SliderAppMaster

public SliderAppMaster()

Service Constructor

Method Detail

serviceInit

public void serviceInit(org.apache.hadoop.conf.Configuration conf)
                 throws Exception

Overrides:

serviceInit in class org.apache.hadoop.service.CompositeService

Throws:

Exception

serviceStart

protected void serviceStart()
                     throws Exception

Overrides:

serviceStart in class org.apache.hadoop.service.CompositeService

Throws:

Exception

bindArgs

public org.apache.hadoop.conf.Configuration bindArgs(org.apache.hadoop.conf.Configuration config,
                                            String... args)
                                              throws Exception

pick up the args from the service launcher

Specified by:

bindArgs in interface RunService

Overrides:

bindArgs in class LaunchedWorkflowCompositeService

Parameters:

config - configuration

args - argument list

Returns:

the configuration

Throws:

Exception - any problem

runService

public int runService()
               throws Throwable

this is called by service launcher; when it returns the application finishes

Specified by:

runService in interface RunService

Overrides:

runService in class LaunchedWorkflowCompositeService

Returns:

the exit code to return by the app

Throws:

Throwable

initAndAddService

public org.apache.hadoop.service.Service initAndAddService(org.apache.hadoop.service.Service service)

Initialize a newly created service then add it. Because the service is not started, this MUST be done before the AM itself starts, or it is explicitly added after

Parameters:

service - the service to init

login

protected void login(String principal,
         File localKeytabFile)
              throws IOException,
                     SliderException

Throws:

IOException

SliderException

validateLoginUser

protected void validateLoginUser(org.apache.hadoop.security.UserGroupInformation user)
                          throws SliderException

Ensure that the user is generated from a keytab and has no HDFS delegation tokens.

Parameters:

user - user to validate

Throws:

SliderException

registerServiceInstance

public void registerServiceInstance(String instanceName,
                           org.apache.hadoop.yarn.api.records.ApplicationId appId)
                             throws IOException

This registers the service instance and its external values

Parameters:

instanceName - name of this instance

appId - application ID

Throws:

IOException

setupInitialRegistryPaths

protected void setupInitialRegistryPaths()
                                  throws IOException

TODO: purge this once RM is doing the work

Throws:

IOException

registerComponent

public boolean registerComponent(org.apache.hadoop.yarn.api.records.ContainerId id,
                        String description)
                          throws IOException

Handler for action Register/re-register an ephemeral container that is already in the app state

Parameters:

id - the component

description - component description

Returns:

true if the component is registered

Throws:

IOException

unregisterComponent

public void unregisterComponent(org.apache.hadoop.yarn.api.records.ContainerId id)

Handler for UnregisterComponentInstance unregister a component. At the time this message is received, the component may not have been registered

Parameters:

id - the component

getLocalConfDir

public File getLocalConfDir()

Build the configuration directory passed in or of the target FS

Returns:

the file

getGeneratedConfDir

public String getGeneratedConfDir()

Get the path to the DFS configuration that is defined in the cluster specification

Returns:

the generated configuration dir

getGlobalInternalOptions

public MapOperations getGlobalInternalOptions()

Get the global internal options for the AM

Returns:

a map to access the internals

getClusterFS

public SliderFileSystem getClusterFS()
                              throws IOException

Get the filesystem of this cluster

Returns:

the FS of the config

Throws:

IOException

getLog

public static org.slf4j.Logger getLog()

Get the AM log

Returns:

the log of the AM

getAppState

public AppState getAppState()

Get the application state

Returns:

the application state

signalAMComplete

public void signalAMComplete(ActionStopSlider stopActionRequest)

Signal that the AM is complete .. queues it in a separate thread

Parameters:

stopActionRequest - request containing shutdown details

onAMStop

public void onAMStop(ActionStopSlider stopActionRequest)

Signal that the AM is complete

Parameters:

stopActionRequest - request containing shutdown details

getProxy

public Object getProxy(Class protocol,
              InetSocketAddress addr)

onContainersAllocated

public void onContainersAllocated(List<org.apache.hadoop.yarn.api.records.Container> allocatedContainers)

Callback event when a container is allocated. The app state is updated with the allocation, and builds up a list of assignments and RM operations. The assignments are handed off into the pool of service launchers to asynchronously schedule container launch operations. The operations are run in sequence; they are expected to be 0 or more release operations (to handle over-allocations)

Specified by:

onContainersAllocated in interface org.apache.hadoop.yarn.client.api.async.AMRMClientAsync.CallbackHandler

Parameters:

allocatedContainers - list of containers that are now ready to be given work.

onContainersCompleted

public void onContainersCompleted(List<org.apache.hadoop.yarn.api.records.ContainerStatus> completedContainers)

Specified by:

onContainersCompleted in interface org.apache.hadoop.yarn.client.api.async.AMRMClientAsync.CallbackHandler

onUpgradeContainers

public void onUpgradeContainers(ActionUpgradeContainers upgradeContainersRequest)
                         throws IOException,
                                SliderException

Signal that containers are being upgraded. Containers specified with --containers option and all containers of all roles specified with --components option are merged and upgraded.

Parameters:

upgradeContainersRequest - request containing upgrade details

Throws:

IOException

SliderException

flexCluster

public void flexCluster(ConfTree resources)
                 throws IOException,
                        SliderException

Implementation of cluster flexing. It should be the only way that anything -even the AM itself on startup- asks for nodes.

Parameters:

resources - the resource tree

Throws:

SliderException - slider problems, including invalid configs

IOException - IO problems

handleReviewAndFlexApplicationSize

public void handleReviewAndFlexApplicationSize(ReviewAndFlexApplicationSize action)
                                        throws SliderInternalStateException

Handle the event requesting a review ... look at the queue and decide whether to act or not

Parameters:

action - action triggering the event. It may be put back into the queue

Throws:

SliderInternalStateException

executeNodeReview

public void executeNodeReview(String reason)
                       throws SliderInternalStateException

Look at where the current node state is -and whether it should be changed

Throws:

SliderInternalStateException

escalateOutstandingRequests

public void escalateOutstandingRequests()

Escalate operation as triggered by external timer.

Get the list of new operations off the AM, then executest them.

onShutdownRequest

public void onShutdownRequest()

RM wants to shut down the AM

Specified by:

onShutdownRequest in interface org.apache.hadoop.yarn.client.api.async.AMRMClientAsync.CallbackHandler

onNodesUpdated

public void onNodesUpdated(List<org.apache.hadoop.yarn.api.records.NodeReport> updatedNodes)

Monitored nodes have been changed

Specified by:

onNodesUpdated in interface org.apache.hadoop.yarn.client.api.async.AMRMClientAsync.CallbackHandler

Parameters:

updatedNodes - list of updated nodes

getProgress

public float getProgress()

heartbeat operation; return the ratio of requested to actual

Specified by:

getProgress in interface org.apache.hadoop.yarn.client.api.async.AMRMClientAsync.CallbackHandler

Returns:

progress

onError

public void onError(Throwable e)

Specified by:

onError in interface org.apache.hadoop.yarn.client.api.async.AMRMClientAsync.CallbackHandler

execute

public void execute(List<AbstractRMOperation> operations)

Description copied from interface: RMOperationHandlerActions

Execute an entire list of operations

Specified by:

execute in interface RMOperationHandlerActions

Parameters:

operations - ops

releaseAssignedContainer

public void releaseAssignedContainer(org.apache.hadoop.yarn.api.records.ContainerId containerId)

Description copied from interface: RMOperationHandlerActions

Release an assigned container

Specified by:

releaseAssignedContainer in interface RMOperationHandlerActions

Parameters:

containerId - container

addContainerRequest

public void addContainerRequest(org.apache.hadoop.yarn.client.api.AMRMClient.ContainerRequest req)

Description copied from interface: RMOperationHandlerActions

Issue a container request

Specified by:

addContainerRequest in interface RMOperationHandlerActions

cancelContainerRequests

public int cancelContainerRequests(org.apache.hadoop.yarn.api.records.Priority priority1,
                          org.apache.hadoop.yarn.api.records.Priority priority2,
                          int count)

Description copied from interface: RMOperationHandlerActions

Remove a container request

Specified by:

cancelContainerRequests in interface RMOperationHandlerActions

Parameters:

priority1 - priority to remove at

priority2 - second priority to target

count - number to remove

cancelSingleRequest

public void cancelSingleRequest(org.apache.hadoop.yarn.client.api.AMRMClient.ContainerRequest request)

Description copied from interface: RMOperationHandlerActions

Cancel a specific request

Specified by:

cancelSingleRequest in interface RMOperationHandlerActions

Parameters:

request - request to cancel

launchProviderService

protected void launchProviderService(AggregateConf instanceDefinition,
                         File confDir)
                              throws IOException,
                                     SliderException

Launch the provider service

Parameters:

instanceDefinition - definition of the service

confDir - directory of config data

Throws:

IOException

SliderException

eventCallbackEvent

public void eventCallbackEvent(Object parameter)

Specified by:

eventCallbackEvent in interface ProviderCompleted

providerLostContainer

public void providerLostContainer(org.apache.hadoop.yarn.api.records.ContainerId containerId)
                           throws SliderException

report container loss. If this isn't already known about, react

Parameters:

containerId - id of the container which has failed

Throws:

SliderException

stateChanged

public void stateChanged(org.apache.hadoop.service.Service service)

Received on listening service termination.

Specified by:

stateChanged in interface org.apache.hadoop.service.ServiceStateChangeListener

Overrides:

stateChanged in class WorkflowCompositeService

Parameters:

service - the service that has changed.

stopForkedProcess

protected Integer stopForkedProcess()

stop forked process if it the running process var is not null

Returns:

the process exit code

startContainer

public void startContainer(org.apache.hadoop.yarn.api.records.Container container,
                  org.apache.hadoop.yarn.api.records.ContainerLaunchContext ctx,
                  RoleInstance instance)
                    throws IOException

Async start container request

Parameters:

container - container

ctx - context

instance - node details

Throws:

IOException

onContainerStopped

public void onContainerStopped(org.apache.hadoop.yarn.api.records.ContainerId containerId)

Specified by:

onContainerStopped in interface org.apache.hadoop.yarn.client.api.async.NMClientAsync.CallbackHandler

onContainerStarted

public void onContainerStarted(org.apache.hadoop.yarn.api.records.ContainerId containerId,
                      Map<String,ByteBuffer> allServiceResponse)

Specified by:

onContainerStarted in interface org.apache.hadoop.yarn.client.api.async.NMClientAsync.CallbackHandler

onStartContainerError

public void onStartContainerError(org.apache.hadoop.yarn.api.records.ContainerId containerId,
                         Throwable t)

Specified by:

onStartContainerError in interface org.apache.hadoop.yarn.client.api.async.NMClientAsync.CallbackHandler

onContainerStatusReceived

public void onContainerStatusReceived(org.apache.hadoop.yarn.api.records.ContainerId containerId,
                             org.apache.hadoop.yarn.api.records.ContainerStatus containerStatus)

Specified by:

onContainerStatusReceived in interface org.apache.hadoop.yarn.client.api.async.NMClientAsync.CallbackHandler

onGetContainerStatusError

public void onGetContainerStatusError(org.apache.hadoop.yarn.api.records.ContainerId containerId,
                             Throwable t)

Specified by:

onGetContainerStatusError in interface org.apache.hadoop.yarn.client.api.async.NMClientAsync.CallbackHandler

onStopContainerError

public void onStopContainerError(org.apache.hadoop.yarn.api.records.ContainerId containerId,
                        Throwable t)

Specified by:

onStopContainerError in interface org.apache.hadoop.yarn.client.api.async.NMClientAsync.CallbackHandler

getInstanceDefinition

public AggregateConf getInstanceDefinition()

getClusterDescription

public ClusterDescription getClusterDescription()

This is the status, the live model

getProviderService

public ProviderService getProviderService()

queue

public void queue(AsyncAction action)

Queue an action for immediate execution in the executor thread

Parameters:

action - action to execute

schedule

public void schedule(AsyncAction action)

Schedule an action

Parameters:

action - for delayed execution

onExceptionInThread

public void onExceptionInThread(Thread thread,
                       Throwable exception)

Handle any exception in a thread. If the exception provides an exit code, that is the one that will be used

Parameters:

thread - thread throwing the exception

exception - exception

main

public static void main(String[] args)

This is the main entry point for the service launcher.

Parameters:

args - command line arguments.