org.apache.slider.client

Class SliderClient

java.lang.Object

org.apache.hadoop.service.AbstractService

org.apache.hadoop.service.CompositeService

org.apache.slider.server.services.workflow.WorkflowCompositeService

org.apache.slider.server.services.utility.LaunchedWorkflowCompositeService

org.apache.slider.server.services.utility.AbstractSliderLaunchedService

org.apache.slider.client.SliderClient

All Implemented Interfaces:

Closeable, AutoCloseable, org.apache.hadoop.service.Service, org.apache.hadoop.service.ServiceStateChangeListener, SliderClientAPI, SliderExitCodes, SliderKeys, SliderXmlConfKeys, ErrorStrings, LauncherExitCodes, RunService, ServiceParent

Direct Known Subclasses:

Slider

public class SliderClient
extends AbstractSliderLaunchedService
implements RunService, SliderExitCodes, SliderKeys, ErrorStrings, SliderClientAPI

Client service for Slider

Nested Class Summary

Nested classes/interfaces inherited from class org.apache.hadoop.service.CompositeService

org.apache.hadoop.service.CompositeService.CompositeServiceShutdownHook

Nested classes/interfaces inherited from interface org.apache.hadoop.service.Service

org.apache.hadoop.service.Service.STATE

Field Summary

Fields

Modifier and Type	Field and Description
org.apache.hadoop.yarn.api.records.ApplicationId	applicationId
static String	E_INSTALL_PATH_DOES_NOT_EXIST
static String	E_INVALID_APPLICATION_PACKAGE_LOCATION
static String	E_INVALID_APPLICATION_TYPE_NAME
static String	E_INVALID_INSTALL_LOCATION
static String	E_INVALID_INSTALL_PATH
static String	E_MUST_BE_A_VALID_JSON_FILE
static String	E_NO_RESOURCE_MANAGER
static String	E_NO_ZOOKEEPER_QUORUM
static String	E_PACKAGE_DOES_NOT_EXIST
static String	E_PACKAGE_EXISTS
static String	E_UNABLE_TO_READ_SUPPLIED_PACKAGE_FILE
static String	E_USE_REPLACEPKG_TO_OVERWRITE
protected SliderFileSystem	sliderFileSystem

Fields inherited from class org.apache.hadoop.service.CompositeService

STOP_ONLY_STARTED_SERVICES

Fields inherited from interface org.apache.slider.common.SliderExitCodes

_EXIT_CODE_BASE, EXIT_APPLICATION_IN_USE, EXIT_BAD_CONFIGURATION, EXIT_BAD_STATE, EXIT_DEPLOYMENT_FAILED, EXIT_INSTANCE_EXISTS, EXIT_PROCESS_FAILED, EXIT_TIMED_OUT, EXIT_UNKNOWN_INSTANCE, EXIT_YARN_SERVICE_FAILED, EXIT_YARN_SERVICE_FINISHED_WITH_ERROR, EXIT_YARN_SERVICE_KILLED

Fields inherited from interface org.apache.slider.core.main.LauncherExitCodes

EXIT_CLIENT_INITIATED_SHUTDOWN, EXIT_COMMAND_ARGUMENT_ERROR, EXIT_CONFLICT, EXIT_CONNECTIVITY_PROBLEM, EXIT_EXCEPTION_THROWN, EXIT_FALSE, EXIT_FORBIDDEN, EXIT_FOUND, EXIT_INTERNAL_ERROR, EXIT_INTERRUPTED, EXIT_MOVED, EXIT_NOT_ACCEPTABLE, EXIT_NOT_FOUND, EXIT_NOT_MODIFIED, EXIT_OPERATION_NOT_ALLOWED, EXIT_OTHER_FAILURE, EXIT_SERVICE_UNAVAILABLE, EXIT_SUCCESS, EXIT_TASK_LAUNCH_FAILURE, EXIT_UNAUTHORIZE, EXIT_UNIMPLEMENTED, EXIT_UNSUPPORTED_VERSION, EXIT_USAGE

Fields inherited from interface org.apache.slider.common.SliderKeys

ADDONS_DIR, AGENT_TAR, AM_DIR_PREFIX, AM_FILTER_NAME, APP_CONTAINER_HEARTBEAT_INTERVAL_SEC, APP_CONTAINER_RELEASE_TIMEOUT, APP_DEF_DIR, APP_TYPE, APP_VERSION, APP_VERSION_UNKNOWN, CLUSTER_DIRECTORY, COMP_KEYSTORE_PASSWORD_ALIAS_DEFAULT, COMP_KEYSTORE_PASSWORD_ALIAS_KEY, COMP_KEYSTORE_PASSWORD_PROPERTY_KEY, COMP_STORES_REQUIRED_KEY, COMP_TRUSTSTORE_PASSWORD_ALIAS_DEFAULT, COMP_TRUSTSTORE_PASSWORD_ALIAS_KEY, COMP_TRUSTSTORE_PASSWORD_PROPERTY_KEY, COMPONENT_AM, CRT_FILE_NAME, CRT_PASS_FILE_NAME, CSR_FILE_NAME, DATA_DIR_NAME, DEFAULT_APP_PKG, DEFAULT_GC_OPTS, DEFAULT_JVM_HEAP, DEFAULT_YARN_MEMORY, GENERATED_CONF_DIR_NAME, GSON_JAR, HADOOP_PROXY_USER, HADOOP_USER_NAME, HDP_VERSION_PROP_NAME, HISTORY_DIR_NAME, HISTORY_FILENAME_CREATION_PATTERN, HISTORY_FILENAME_GLOB_PATTERN, HISTORY_FILENAME_MATCH_PATTERN, HISTORY_FILENAME_PREFIX, HISTORY_FILENAME_SUFFIX, INFRA_DIR_NAME, JCOMMANDER_JAR, JVM_ENABLE_ASSERTIONS, JVM_ENABLE_SYSTEM_ASSERTIONS, JVM_FORCE_IPV4, JVM_JAVA_HEADLESS, KEY_ALLOWED_PORT_RANGE, KEY_AM_ALLOWED_PORT_RANGE, KEY_FILE_NAME, KEYSTORE_FILE_NAME, KEYTAB_DIR, LOCAL_TARBALL_INSTALL_SUBDIR, LOG4J_PROP_FILENAME, LOG4J_SERVER_PROP_FILENAME, PACKAGE_DIRECTORY, PASS_LEN, PROPAGATE_RESOURCE_OPTION, PROPAGATED_CONF_DIR_NAME, PROPERTY_CONF_DIR, PROPERTY_LIB_DIR, PYTHONPATH, ROLE_AM_PRIORITY_INDEX, SECURITY_DIR, SLIDER_BASE_DIRECTORY, SLIDER_CLASSPATH_EXTRA, SLIDER_CLIENT_XML, SLIDER_DEPENDENCY_DIR, SLIDER_DEPENDENCY_DIR_PERMISSIONS, SLIDER_DEPENDENCY_HDP_PARENT_DIR, SLIDER_DEPENDENCY_LOCALIZED_DIR_LINK, SLIDER_DEPENDENCY_TAR_GZ_FILE_EXT, SLIDER_DEPENDENCY_TAR_GZ_FILE_NAME, SLIDER_JAR, SLIDER_JVM_OPTS, SLIDER_PASSPHRASE, SLIDER_SERVER_XML, SLIDER_XML, SNAPSHOT_CONF_DIR_NAME, STDERR_AM, STDOUT_AM, SUBMITTED_CONF_DIR, SYSPROP_LOG_DIR, SYSPROP_LOG4J_CONFIGURATION, TMP_DIR_PREFIX, TMP_LOGDIR_PREFIX, YARN_CONTAINER_PATH

Fields inherited from interface org.apache.slider.common.SliderXmlConfKeys

CLUSTER_DIRECTORY_PERMISSIONS, DATA_DIRECTORY_PERMISSIONS, DEFAULT_CLUSTER_DIRECTORY_PERMISSIONS, DEFAULT_DATA_DIRECTORY_PERMISSIONS, DEFAULT_YARN_QUEUE, DEFAULT_YARN_QUEUE_PRIORITY, DFS_DATANODE_KERBEROS_PRINCIPAL_KEY, DFS_NAMENODE_DELEGATION_KEY_UPDATE_INTERVAL_DEFAULT, DFS_NAMENODE_DELEGATION_KEY_UPDATE_INTERVAL_KEY, DFS_NAMENODE_DELEGATION_TOKEN_ALWAYS_USE_DEFAULT, DFS_NAMENODE_DELEGATION_TOKEN_ALWAYS_USE_KEY, DFS_NAMENODE_DELEGATION_TOKEN_MAX_LIFETIME_DEFAULT, DFS_NAMENODE_DELEGATION_TOKEN_MAX_LIFETIME_KEY, DFS_NAMENODE_DELEGATION_TOKEN_RENEW_INTERVAL_DEFAULT, DFS_NAMENODE_DELEGATION_TOKEN_RENEW_INTERVAL_KEY, DFS_NAMENODE_DU_RESERVED_KEY, DFS_NAMENODE_KERBEROS_PRINCIPAL_KEY, DFS_NAMENODE_KEYTAB_FILE_KEY, FS_DEFAULT_NAME_CLASSIC, HADOOP_HTTP_FILTER_INITIALIZERS, IPC_CLIENT_FALLBACK_TO_SIMPLE_AUTH, IPC_CLIENT_RETRY_POLICY_ENABLED_DEFAULT, IPC_CLIENT_RETRY_POLICY_SPEC_DEFAULT, KEY_AM_KEYTAB_LOCAL_PATH, KEY_AM_LAUNCH_ENV, KEY_AM_LOGIN_KEYTAB_NAME, KEY_AM_LOGIN_KEYTAB_REQUIRED, KEY_AM_RESTART_LIMIT, KEY_HDFS_KEYTAB_DIR, KEY_IPC_CLIENT_RETRY_POLICY_ENABLED, KEY_IPC_CLIENT_RETRY_POLICY_SPEC, KEY_KERBEROS_PRINCIPAL, KEY_KEYSTORE_LOCATION, KEY_KEYTAB_PRINCIPAL, KEY_PROTOCOL_ACL, KEY_PROVIDER, KEY_SECURITY_ENABLED, KEY_SLIDER_AM_DEPENDENCY_CHECKS_DISABLED, KEY_SLIDER_BASE_PATH, KEY_TEMPLATE_ORIGIN, KEY_YARN_QUEUE, KEY_YARN_QUEUE_PRIORITY, PREFIX_PROVIDER, PYTHON_EXECUTABLE_PATH, REGISTRY_PATH, REGISTRY_ZK_QUORUM, X_DEV_INSECURE_DEFAULT, X_DEV_INSECURE_REQUIRED, X_DEV_INSECURE_WS

Fields inherited from interface org.apache.slider.core.exceptions.ErrorStrings

E_ALREADY_EXISTS, E_APPLICATION_NOT_RUNNING, E_BOTH_IMAGE_AND_HOME_DIR_SPECIFIED, E_CLUSTER_RUNNING, E_CONFIGURATION_DIRECTORY_NOT_FOUND, E_DESTROY_CREATE_RACE_CONDITION, E_FINISHED_APPLICATION, E_INCOMPLETE_CLUSTER_SPEC, E_MISSING_PATH, E_NO_IMAGE_OR_HOME_DIR_SPECIFIED, E_UNKNOWN_INSTANCE, E_UNKNOWN_ROLE, E_UNSTABLE_CLUSTER, ERROR_DUPLICATE_ENTRY, ERROR_NO_ACTION, ERROR_NOT_ENOUGH_ARGUMENTS, ERROR_PARSE_FAILURE, ERROR_TOO_MANY_ARGUMENTS, ERROR_UNKNOWN_ACTION, PRINTF_E_INSTANCE_ALREADY_EXISTS, PRINTF_E_INSTANCE_DIR_ALREADY_EXISTS

Constructor Summary

Constructors

Constructor and Description
SliderClient() Constructor

Method Summary

Methods

Modifier and Type	Method and Description
int	actionAmSuicide(String clustername, ActionAMSuicideArgs args) AM to commit an asynchronous suicide
int	actionBuild(String clustername, AbstractClusterBuildingActionArgs buildInfo) Build up the cluster specification/directory
int	actionClient(ActionClientArgs clientInfo) Perform client operations such as install or configure
int	actionCreate(String clustername, ActionCreateArgs createArgs) Create the cluster -saving the arguments to a specification file first
int	actionDependency(ActionDependencyArgs args) Upload all Slider AM and agent dependency libraries to HDFS, so that they do not need to be uploaded with every create call.
int	actionDestroy(String clustername) Keep this signature for backward compatibility with force=true by default.
int	actionDestroy(String clustername, ActionDestroyArgs destroyArgs) Destroy a cluster.
int	actionDiagnostic(ActionDiagnosticArgs diagnosticArgs) diagnostic operation
String	actionEcho(String name, ActionEchoArgs args) Echo operation (not currently wired up to command line)
int	actionExists(String name, ActionExistsArgs args)
int	actionExists(String name, boolean checkLive) Test for a cluster existing probe for a cluster of the given name existing in the filesystem.
int	actionFlex(String name, ActionFlexArgs args) Implement the islive action: probe for a cluster of the given name existing
int	actionFreeze(String clustername, ActionFreezeArgs freezeArgs) Stop the cluster
int	actionInstallKeytab(ActionInstallKeytabArgs installKeytabInfo) Upload keytab to a designated sub-directory of the user home directory
int	actionInstallPkg(ActionInstallPackageArgs installPkgInfo) Upload application package to user home directory
int	actionKeytab(ActionKeytabArgs keytabInfo) Manage keytabs leveraged by slider
int	actionKillContainer(String name, ActionKillContainerArgs args) Kill a specific container of the cluster
int	actionList(String clustername) A basic list action to list live instances
int	actionList(String clustername, ActionListArgs args) Implement the list action.
int	actionLookup(ActionLookupArgs args) Implement the lookup action.
int	actionNodes(String instance, ActionNodesArgs args) List the nodes in the cluster, possibly filtering by node state or label.
int	actionPackage(ActionPackageArgs actionPackageInfo) Managing slider application package
int	actionRegistry(ActionRegistryArgs registryArgs) Registry operation
PublishedConfiguration	actionRegistryGetConfig(ActionRegistryArgs registryArgs) list configs available for an instance
PublishedExports	actionRegistryGetExport(ActionRegistryArgs registryArgs) get a specific export group
Collection<org.apache.hadoop.registry.client.types.ServiceRecord>	actionRegistryList(ActionRegistryArgs registryArgs) Registry operation
void	actionRegistryListConfigsYarn(ActionRegistryArgs registryArgs) list configs available for an instance
void	actionRegistryListExports(ActionRegistryArgs registryArgs) list exports available for an instance
int	actionResolve(ActionResolveArgs args) Registry operation
int	actionStatus(String clustername, ActionStatusArgs statusArgs) Status operation
int	actionThaw(String clustername, ActionThawArgs thaw) Restore a cluster
int	actionUpdate(String clustername, AbstractClusterBuildingActionArgs buildInfo) Update the cluster specification
int	actionUpgrade(String clustername, ActionUpgradeArgs upgradeArgs) Upgrade the cluster with a newer version of the application
int	actionVersion() Version Details
org.apache.hadoop.conf.Configuration	bindArgs(org.apache.hadoop.conf.Configuration config, String... args) This is called Before serviceInit is called
void	buildInstanceDefinition(String clustername, AbstractClusterBuildingActionArgs buildInfo, boolean overwrite, boolean liveClusterAllowed) Build up the AggregateConfiguration for an application instance then persists it
void	buildInstanceDefinition(String clustername, AbstractClusterBuildingActionArgs buildInfo, boolean overwrite, boolean liveClusterAllowed, boolean isUpgradeFlow)
AbstractClientProvider	createClientProvider(String provider) Get the provider for this cluster
SliderApplicationApi	createIpcClient() Create a new IPC client for talking to slider via what follows the REST API.
org.apache.hadoop.fs.Path	createPathThatMustExist(String uri) Create a path that must exist in the cluster fs
String	createZookeeperNode(String clusterName, Boolean nameOnly) Create the zookeeper node associated with the calling user and the cluster
String	createZookeeperNodeInner(String clusterName, Boolean nameOnly) Create the zookeeper node associated with the calling user and the cluster -throwing exceptions on any failure
boolean	deleteZookeeperNode(String clusterName) Delete the zookeeper node associated with the calling user and the cluster TODO: YARN registry operations
Map<String,SliderInstanceDescription>	enumSliderInstances(boolean listOnlyInState, org.apache.hadoop.yarn.api.records.YarnApplicationState minAppState, org.apache.hadoop.yarn.api.records.YarnApplicationState maxAppState) Enumerate slider instances for the current user, and the most recent app report, where available.
int	exec() Execute the command line
int	flex(String clustername, Map<String,Integer> roleInstances) Implement flexing
boolean	forceKillApplication(String reason) Kill the submitted application via YARN
protected Map<String,String>	generatePlaceholderKeyValueMap(String env)
protected Map<String,String>	getAmLaunchEnv(org.apache.hadoop.conf.Configuration config)
Set<String>	getApplicationList(String clustername) Retrieve a list of all live instances.
Set<String>	getApplicationList(String clustername, ActionListArgs args) Retrieve a list of application instances satisfying the query criteria.
org.apache.hadoop.yarn.api.records.ApplicationReport	getApplicationReport() Get the report of a this application
org.apache.hadoop.yarn.api.records.ApplicationReport	getApplicationReport(org.apache.hadoop.yarn.api.records.ApplicationId appId)
List<org.apache.hadoop.yarn.api.records.ApplicationReport>	getApplications() Get all YARN applications
ClusterDescription	getClusterDescription() Connect to the cluster and get its current state
ClusterDescription	getClusterDescription(String clustername) Connect to a live cluster and get its current state
org.apache.hadoop.fs.permission.FsPermission	getClusterDirectoryPermissions(org.apache.hadoop.conf.Configuration conf)
List<ContainerInformation>	getContainers(String name)
String	getDeployedClusterName() Get the name of any deployed cluster
AggregateConf	getLaunchedInstanceDefinition() The configuration used for deployment (after resolution).
AggregateConf	getLiveInstanceDefinition() Get the instance definition from the far end
org.apache.hadoop.registry.client.api.RegistryOperations	getRegistryOperations() Get the registry binding.
String	getUsername()
YarnAppListClient	getYarnAppListClient() Get at the service registry operations
protected ZKIntegration	getZkClient(String clusterName, String user) Gets a zookeeper client, returns null if it cannot connect to zookeeper
protected void	initHadoopBinding() Perform everything needed to init the hadoop binding.
boolean	isApplicationActive(org.apache.hadoop.yarn.api.records.ApplicationReport report) Is an application active: accepted or running
LaunchedApplication	launchApplication(String clustername, org.apache.hadoop.fs.Path clusterDirectory, AggregateConf instanceDefinition, boolean debugAM)
List<ClusterNode>	listClusterNodes(String[] uuids) Get the details on a list of uuids
List<ClusterNode>	listClusterNodesInRole(String role) List all nodes in a role.
NodeInformationList	listInstanceNodes(String instance, ActionNodesArgs args) List the nodes in the cluster, possibly filtering by node state or label.
String[]	listNodeUUIDsByRole(String role) List all node UUIDs in a role
List<String>	listRegisteredSliderInstances() List instances in the registry
Map<String,org.apache.hadoop.registry.client.types.ServiceRecord>	listRegistryInstances() List instances in the registry for the current user
List<org.apache.hadoop.yarn.api.records.ApplicationReport>	listSliderInstances(String user) List Slider instances belonging to a specific user.
NodeInformationList	listYarnClusterNodes(ActionNodesArgs args) List the nodes in the cluster, possibly filtering by node state or label.
AggregateConf	loadInstanceDefinition(String name, boolean resolved) Load the instance definition.
AggregateConf	loadInstanceDefinitionUnresolved(String name, org.apache.hadoop.fs.Path clusterDirectory) Load the instance definition.
AggregateConf	loadPersistedClusterDescription(String clustername) Load the persistent cluster description
org.apache.hadoop.registry.client.types.ServiceRecord	lookupServiceRecord(String serviceType, String id) Look up a service record of the current user
org.apache.hadoop.yarn.api.records.ApplicationReport	monitorAppToRunning(Duration duration) Wait for the app to start running (or go past that state)
org.apache.hadoop.yarn.api.records.ApplicationReport	monitorAppToState(org.apache.hadoop.yarn.api.records.YarnApplicationState desiredState, Duration duration) Monitor the submitted application for reaching the requested state.
protected void	persistInstanceDefinition(boolean overwrite, org.apache.hadoop.fs.Path appconfdir, InstanceBuilder builder)
static void	replaceTokens(ConfTree conf, String userName, String clusterName)
org.apache.hadoop.registry.client.types.ServiceRecord	resolve(String path) Look up an instance
int	runService() Launched service execution.
protected void	serviceInit(org.apache.hadoop.conf.Configuration conf)
void	setDeployedClusterName(String deployedClusterName)
String	toString()
UnknownApplicationInstanceException	unknownClusterException(String clustername) Generate an exception for an unknown cluster
protected void	validateInstanceDefinition(AbstractClientProvider provider, AggregateConf instanceDefinition, SliderFileSystem fs) Validate an instance definition against a provider.
void	verifyBindingsDefined() Verify that the Resource Manager is configured (on a non-HA cluster).
void	verifyNoLiveClusters(String clustername, String action) verify that a live cluster isn't there
int	waitForAppRunning(LaunchedApplication launchedApplication, int acceptWaitMillis, int runWaitMillis) Wait for the launched app to be accepted in the time and, optionally running.
int	waitForRoleInstanceLive(String role, long timeout) Wait for an instance of a named role to be live (or past it in the lifecycle)

Methods inherited from class org.apache.slider.server.services.utility.AbstractSliderLaunchedService

createRegistryOperationsInstance, lookupZKQuorum, require, requireArgumentSet, startRegistryOperationsService

Methods inherited from class org.apache.slider.server.services.utility.LaunchedWorkflowCompositeService

addService, deployChildService, getArgv, isSet, isUnset

Methods inherited from class org.apache.slider.server.services.workflow.WorkflowCompositeService

stateChanged

Methods inherited from class org.apache.hadoop.service.CompositeService

addIfService, getServices, removeService, serviceStart, serviceStop

Methods inherited from class org.apache.hadoop.service.AbstractService

close, getBlockers, getConfig, getFailureCause, getFailureState, getLifecycleHistory, getName, getServiceState, getStartTime, init, isInState, noteFailure, putBlocker, registerGlobalListener, registerServiceListener, removeBlocker, setConfig, start, stop, unregisterGlobalListener, unregisterServiceListener, waitForServiceToStop

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

Methods inherited from interface org.apache.hadoop.service.Service

close, getBlockers, getConfig, getFailureCause, getFailureState, getLifecycleHistory, getName, getServiceState, getStartTime, init, isInState, registerServiceListener, start, stop, unregisterServiceListener, waitForServiceToStop

Methods inherited from interface org.apache.slider.server.services.workflow.ServiceParent

getServices

Field Detail

E_MUST_BE_A_VALID_JSON_FILE

public static final String E_MUST_BE_A_VALID_JSON_FILE

See Also:

Constant Field Values

E_INVALID_INSTALL_LOCATION

public static final String E_INVALID_INSTALL_LOCATION

See Also:

Constant Field Values

E_UNABLE_TO_READ_SUPPLIED_PACKAGE_FILE

public static final String E_UNABLE_TO_READ_SUPPLIED_PACKAGE_FILE

See Also:

Constant Field Values

E_INVALID_APPLICATION_PACKAGE_LOCATION

public static final String E_INVALID_APPLICATION_PACKAGE_LOCATION

See Also:

Constant Field Values

E_INVALID_INSTALL_PATH

public static final String E_INVALID_INSTALL_PATH

See Also:

Constant Field Values

E_INSTALL_PATH_DOES_NOT_EXIST

public static final String E_INSTALL_PATH_DOES_NOT_EXIST

See Also:

Constant Field Values

E_INVALID_APPLICATION_TYPE_NAME

public static final String E_INVALID_APPLICATION_TYPE_NAME

See Also:

Constant Field Values

E_USE_REPLACEPKG_TO_OVERWRITE

public static final String E_USE_REPLACEPKG_TO_OVERWRITE

See Also:

Constant Field Values

E_PACKAGE_DOES_NOT_EXIST

public static final String E_PACKAGE_DOES_NOT_EXIST

See Also:

Constant Field Values

E_NO_ZOOKEEPER_QUORUM

public static final String E_NO_ZOOKEEPER_QUORUM

See Also:

Constant Field Values

E_NO_RESOURCE_MANAGER

public static final String E_NO_RESOURCE_MANAGER

See Also:

Constant Field Values

E_PACKAGE_EXISTS

public static final String E_PACKAGE_EXISTS

See Also:

Constant Field Values

applicationId

public org.apache.hadoop.yarn.api.records.ApplicationId applicationId

sliderFileSystem

protected SliderFileSystem sliderFileSystem

Constructor Detail

SliderClient

public SliderClient()

Constructor

Method Detail

bindArgs

public org.apache.hadoop.conf.Configuration bindArgs(org.apache.hadoop.conf.Configuration config,
                                            String... args)
                                              throws Exception

This is called Before serviceInit is called

Specified by:

bindArgs in interface RunService

Overrides:

bindArgs in class LaunchedWorkflowCompositeService

Parameters:

config - the initial configuration build up by the service launcher.

args - argument list list of arguments passed to the command line after any launcher-specific commands have been stripped.

Returns:

the post-binding configuration to pass to the init() operation.

Throws:

Exception

serviceInit

protected void serviceInit(org.apache.hadoop.conf.Configuration conf)
                    throws Exception

Overrides:

serviceInit in class org.apache.hadoop.service.CompositeService

Throws:

Exception

runService

public int runService()
               throws Throwable

Launched service execution. This runs exec() then catches some exceptions and converts them to exit codes

Specified by:

runService in interface RunService

Overrides:

runService in class LaunchedWorkflowCompositeService

Returns:

an exit code

Throws:

Throwable

exec

public int exec()
         throws Throwable

Execute the command line

Returns:

an exit code

Throws:

Throwable - on a failure

initHadoopBinding

protected void initHadoopBinding()
                          throws IOException,
                                 SliderException

Perform everything needed to init the hadoop binding. This assumes that the service is already in inited or started state

Throws:

IOException

SliderException

deleteZookeeperNode

public boolean deleteZookeeperNode(String clusterName)
                            throws org.apache.hadoop.yarn.exceptions.YarnException,
                                   IOException

Delete the zookeeper node associated with the calling user and the cluster TODO: YARN registry operations

Throws:

org.apache.hadoop.yarn.exceptions.YarnException

IOException

createZookeeperNode

public String createZookeeperNode(String clusterName,
                         Boolean nameOnly)
                           throws org.apache.hadoop.yarn.exceptions.YarnException,
                                  IOException

Create the zookeeper node associated with the calling user and the cluster

Parameters:

clusterName - slider application name

nameOnly - should the name only be created (i.e. don't create ZK node)

Returns:

the path, using the policy implemented in ZKIntegration.mkClusterPath(String, String)

Throws:

org.apache.hadoop.yarn.exceptions.YarnException

IOException

createZookeeperNodeInner

public String createZookeeperNodeInner(String clusterName,
                              Boolean nameOnly)
                                throws org.apache.hadoop.yarn.exceptions.YarnException,
                                       IOException,
                                       org.apache.zookeeper.KeeperException,
                                       InterruptedException

Create the zookeeper node associated with the calling user and the cluster -throwing exceptions on any failure

Parameters:

clusterName - cluster name

nameOnly - create the path, not the node

Returns:

the path, using the policy implemented in ZKIntegration.mkClusterPath(String, String)

Throws:

org.apache.hadoop.yarn.exceptions.YarnException

IOException

org.apache.zookeeper.KeeperException

InterruptedException

getZkClient

protected ZKIntegration getZkClient(String clusterName,
                        String user)
                             throws org.apache.hadoop.yarn.exceptions.YarnException

Gets a zookeeper client, returns null if it cannot connect to zookeeper

Throws:

org.apache.hadoop.yarn.exceptions.YarnException

actionDestroy

public int actionDestroy(String clustername)
                  throws org.apache.hadoop.yarn.exceptions.YarnException,
                         IOException

Keep this signature for backward compatibility with force=true by default.

Specified by:

actionDestroy in interface SliderClientAPI

Throws:

org.apache.hadoop.yarn.exceptions.YarnException

IOException

actionDestroy

public int actionDestroy(String clustername,
                ActionDestroyArgs destroyArgs)
                  throws org.apache.hadoop.yarn.exceptions.YarnException,
                         IOException

Description copied from interface: SliderClientAPI

Destroy a cluster. There's two race conditions here #1 the cluster is started between verifying that there are no live clusters of that name.

Specified by:

actionDestroy in interface SliderClientAPI

Throws:

org.apache.hadoop.yarn.exceptions.YarnException

IOException

actionAmSuicide

public int actionAmSuicide(String clustername,
                  ActionAMSuicideArgs args)
                    throws org.apache.hadoop.yarn.exceptions.YarnException,
                           IOException

Description copied from interface: SliderClientAPI

AM to commit an asynchronous suicide

Specified by:

actionAmSuicide in interface SliderClientAPI

Throws:

org.apache.hadoop.yarn.exceptions.YarnException

IOException

createClientProvider

public AbstractClientProvider createClientProvider(String provider)
                                            throws SliderException

Description copied from interface: SliderClientAPI

Get the provider for this cluster

Specified by:

createClientProvider in interface SliderClientAPI

Parameters:

provider - the name of the provider

Returns:

the provider instance

Throws:

SliderException - problems building the provider

actionCreate

public int actionCreate(String clustername,
               ActionCreateArgs createArgs)
                 throws org.apache.hadoop.yarn.exceptions.YarnException,
                        IOException

Create the cluster -saving the arguments to a specification file first

Parameters:

clustername - cluster name

Returns:

the status code

Throws:

org.apache.hadoop.yarn.exceptions.YarnException - Yarn problems

IOException - other problems

BadCommandArgumentsException - bad arguments.

actionUpgrade

public int actionUpgrade(String clustername,
                ActionUpgradeArgs upgradeArgs)
                  throws org.apache.hadoop.yarn.exceptions.YarnException,
                         IOException

Description copied from interface: SliderClientAPI

Upgrade the cluster with a newer version of the application

Specified by:

actionUpgrade in interface SliderClientAPI

Parameters:

clustername - cluster name

upgradeArgs - the arguments needed to upgrade the cluster

Throws:

org.apache.hadoop.yarn.exceptions.YarnException - Yarn problems

IOException - other problems

actionBuild

public int actionBuild(String clustername,
              AbstractClusterBuildingActionArgs buildInfo)
                throws org.apache.hadoop.yarn.exceptions.YarnException,
                       IOException

Description copied from interface: SliderClientAPI

Build up the cluster specification/directory

Specified by:

actionBuild in interface SliderClientAPI

Parameters:

clustername - cluster name

buildInfo - the arguments needed to build the cluster

Throws:

org.apache.hadoop.yarn.exceptions.YarnException - Yarn problems

BadCommandArgumentsException - bad arguments.

IOException - other problems

actionKeytab

public int actionKeytab(ActionKeytabArgs keytabInfo)
                 throws org.apache.hadoop.yarn.exceptions.YarnException,
                        IOException

Description copied from interface: SliderClientAPI

Manage keytabs leveraged by slider

Specified by:

actionKeytab in interface SliderClientAPI

Parameters:

keytabInfo - the arguments needed to manage the keytab

Throws:

org.apache.hadoop.yarn.exceptions.YarnException - Yarn problems

BadCommandArgumentsException - bad arguments.

IOException - other problems

actionInstallKeytab

public int actionInstallKeytab(ActionInstallKeytabArgs installKeytabInfo)
                        throws org.apache.hadoop.yarn.exceptions.YarnException,
                               IOException

Description copied from interface: SliderClientAPI

Upload keytab to a designated sub-directory of the user home directory

Specified by:

actionInstallKeytab in interface SliderClientAPI

Parameters:

installKeytabInfo - the arguments needed to upload the keytab

Throws:

org.apache.hadoop.yarn.exceptions.YarnException - Yarn problems

BadCommandArgumentsException - bad arguments.

IOException - other problems

actionInstallPkg

public int actionInstallPkg(ActionInstallPackageArgs installPkgInfo)
                     throws org.apache.hadoop.yarn.exceptions.YarnException,
                            IOException

Description copied from interface: SliderClientAPI

Upload application package to user home directory

Specified by:

actionInstallPkg in interface SliderClientAPI

Parameters:

installPkgInfo - the arguments needed to upload the package

Throws:

org.apache.hadoop.yarn.exceptions.YarnException - Yarn problems

BadCommandArgumentsException - bad arguments.

IOException - other problems

actionClient

public int actionClient(ActionClientArgs clientInfo)
                 throws org.apache.hadoop.yarn.exceptions.YarnException,
                        IOException

Description copied from interface: SliderClientAPI

Perform client operations such as install or configure

Specified by:

actionClient in interface SliderClientAPI

Parameters:

clientInfo - the arguments needed for client operations

Throws:

SliderException - bad arguments.

IOException - problems related to package and destination folders

org.apache.hadoop.yarn.exceptions.YarnException

actionPackage

public int actionPackage(ActionPackageArgs actionPackageInfo)
                  throws org.apache.hadoop.yarn.exceptions.YarnException,
                         IOException

Description copied from interface: SliderClientAPI

Managing slider application package

Specified by:

actionPackage in interface SliderClientAPI

Parameters:

actionPackageInfo - the arguments needed to upload, delete or list the package

Throws:

org.apache.hadoop.yarn.exceptions.YarnException - Yarn problems

BadCommandArgumentsException - bad arguments.

IOException - other problems

actionUpdate

public int actionUpdate(String clustername,
               AbstractClusterBuildingActionArgs buildInfo)
                 throws org.apache.hadoop.yarn.exceptions.YarnException,
                        IOException

Description copied from interface: SliderClientAPI

Update the cluster specification

Specified by:

actionUpdate in interface SliderClientAPI

Parameters:

clustername - cluster name

buildInfo - the arguments needed to update the cluster

Throws:

org.apache.hadoop.yarn.exceptions.YarnException - Yarn problems

IOException - other problems

buildInstanceDefinition

public void buildInstanceDefinition(String clustername,
                           AbstractClusterBuildingActionArgs buildInfo,
                           boolean overwrite,
                           boolean liveClusterAllowed)
                             throws org.apache.hadoop.yarn.exceptions.YarnException,
                                    IOException

Build up the AggregateConfiguration for an application instance then persists it

Parameters:

clustername - name of the cluster

buildInfo - the arguments needed to build the cluster

overwrite - true if existing cluster directory can be overwritten

liveClusterAllowed - true if live cluster can be modified

Throws:

org.apache.hadoop.yarn.exceptions.YarnException

IOException

buildInstanceDefinition

public void buildInstanceDefinition(String clustername,
                           AbstractClusterBuildingActionArgs buildInfo,
                           boolean overwrite,
                           boolean liveClusterAllowed,
                           boolean isUpgradeFlow)
                             throws org.apache.hadoop.yarn.exceptions.YarnException,
                                    IOException

Throws:

org.apache.hadoop.yarn.exceptions.YarnException

IOException

persistInstanceDefinition

protected void persistInstanceDefinition(boolean overwrite,
                             org.apache.hadoop.fs.Path appconfdir,
                             InstanceBuilder builder)
                                  throws IOException,
                                         SliderException,
                                         LockAcquireFailedException

Throws:

IOException

SliderException

LockAcquireFailedException

replaceTokens

public static void replaceTokens(ConfTree conf,
                 String userName,
                 String clusterName)
                          throws IOException

Throws:

IOException

getClusterDirectoryPermissions

public org.apache.hadoop.fs.permission.FsPermission getClusterDirectoryPermissions(org.apache.hadoop.conf.Configuration conf)

verifyBindingsDefined

public void verifyBindingsDefined()
                           throws BadCommandArgumentsException

Verify that the Resource Manager is configured (on a non-HA cluster). with a useful error message

Throws:

BadCommandArgumentsException - the exception raised on an invalid config

loadInstanceDefinitionUnresolved

public AggregateConf loadInstanceDefinitionUnresolved(String name,
                                             org.apache.hadoop.fs.Path clusterDirectory)
                                               throws IOException,
                                                      SliderException

Load the instance definition. It is not resolved at this point

Parameters:

name - cluster name

clusterDirectory - cluster dir

Returns:

the loaded configuration

Throws:

IOException

SliderException

UnknownApplicationInstanceException - if the file is not found

loadInstanceDefinition

public AggregateConf loadInstanceDefinition(String name,
                                   boolean resolved)
                                     throws IOException,
                                            SliderException

Load the instance definition.

Parameters:

name - cluster name

resolved - flag to indicate the cluster should be resolved

Returns:

the loaded configuration

Throws:

IOException - IO problems

SliderException - slider explicit issues

UnknownApplicationInstanceException - if the file is not found

launchApplication

public LaunchedApplication launchApplication(String clustername,
                                    org.apache.hadoop.fs.Path clusterDirectory,
                                    AggregateConf instanceDefinition,
                                    boolean debugAM)
                                      throws org.apache.hadoop.yarn.exceptions.YarnException,
                                             IOException

Parameters:

clustername - name of the cluster

clusterDirectory - cluster dir

instanceDefinition - the instance definition

debugAM - enable debug AM options

Returns:

the launched application

Throws:

org.apache.hadoop.yarn.exceptions.YarnException

IOException

getAmLaunchEnv

protected Map<String,String> getAmLaunchEnv(org.apache.hadoop.conf.Configuration config)

generatePlaceholderKeyValueMap

protected Map<String,String> generatePlaceholderKeyValueMap(String env)

waitForAppRunning

public int waitForAppRunning(LaunchedApplication launchedApplication,
                    int acceptWaitMillis,
                    int runWaitMillis)
                      throws org.apache.hadoop.yarn.exceptions.YarnException,
                             IOException

Wait for the launched app to be accepted in the time and, optionally running.

If the application

Parameters:

launchedApplication - application

acceptWaitMillis - time in millis to wait for accept

runWaitMillis - time in millis to wait for the app to be running. May be null, in which case no wait takes place

Returns:

exit code: success

Throws:

org.apache.hadoop.yarn.exceptions.YarnException

IOException

createPathThatMustExist

public org.apache.hadoop.fs.Path createPathThatMustExist(String uri)
                                                  throws SliderException,
                                                         IOException

Create a path that must exist in the cluster fs

Parameters:

uri - uri to create

Returns:

the path

Throws:

FileNotFoundException - if the path does not exist

SliderException

IOException

verifyNoLiveClusters

public void verifyNoLiveClusters(String clustername,
                        String action)
                          throws IOException,
                                 org.apache.hadoop.yarn.exceptions.YarnException

verify that a live cluster isn't there

Parameters:

clustername - cluster name

action -

Throws:

SliderException - with exit code EXIT_CLUSTER_LIVE if a cluster of that name is either live or starting up.

IOException

org.apache.hadoop.yarn.exceptions.YarnException

getUsername

public String getUsername()
                   throws IOException

Throws:

IOException

getDeployedClusterName

public String getDeployedClusterName()

Get the name of any deployed cluster

Returns:

the cluster name

setDeployedClusterName

public void setDeployedClusterName(String deployedClusterName)

monitorAppToRunning

public org.apache.hadoop.yarn.api.records.ApplicationReport monitorAppToRunning(Duration duration)
                                                                         throws org.apache.hadoop.yarn.exceptions.YarnException,
                                                                                IOException

Wait for the app to start running (or go past that state)

Parameters:

duration - time to wait

Returns:

the app report; null if the duration turned out

Throws:

org.apache.hadoop.yarn.exceptions.YarnException - YARN or app issues

IOException - IO problems

monitorAppToState

public org.apache.hadoop.yarn.api.records.ApplicationReport monitorAppToState(org.apache.hadoop.yarn.api.records.YarnApplicationState desiredState,
                                                                     Duration duration)
                                                                       throws org.apache.hadoop.yarn.exceptions.YarnException,
                                                                              IOException

Monitor the submitted application for reaching the requested state. Will also report if the app reaches a later state (failed, killed, etc) Kill application if duration!= null & time expires. Prerequisite: the applicatin was launched.

Parameters:

desiredState - desired state.

duration - how long to wait -must be more than 0

Returns:

the application report -null on a timeout

Throws:

org.apache.hadoop.yarn.exceptions.YarnException

IOException

getApplicationReport

public org.apache.hadoop.yarn.api.records.ApplicationReport getApplicationReport()
                                                                          throws IOException,
                                                                                 org.apache.hadoop.yarn.exceptions.YarnException

Description copied from interface: SliderClientAPI

Get the report of a this application

Specified by:

getApplicationReport in interface SliderClientAPI

Returns:

the app report or null if it could not be found.

Throws:

IOException

org.apache.hadoop.yarn.exceptions.YarnException

forceKillApplication

public boolean forceKillApplication(String reason)
                             throws org.apache.hadoop.yarn.exceptions.YarnException,
                                    IOException

Description copied from interface: SliderClientAPI

Kill the submitted application via YARN

Specified by:

forceKillApplication in interface SliderClientAPI

Throws:

org.apache.hadoop.yarn.exceptions.YarnException

IOException

listSliderInstances

public List<org.apache.hadoop.yarn.api.records.ApplicationReport> listSliderInstances(String user)
                                                                               throws org.apache.hadoop.yarn.exceptions.YarnException,
                                                                                      IOException

List Slider instances belonging to a specific user. This will include failed and killed instances; there may be duplicates

Parameters:

user - user: "" means all users, null means "default"

Returns:

a possibly empty list of Slider AMs

Throws:

org.apache.hadoop.yarn.exceptions.YarnException

IOException

actionList

public int actionList(String clustername)
               throws IOException,
                      org.apache.hadoop.yarn.exceptions.YarnException

A basic list action to list live instances

Parameters:

clustername - cluster name

Returns:

success if the listing was considered successful

Throws:

IOException

org.apache.hadoop.yarn.exceptions.YarnException

actionList

public int actionList(String clustername,
             ActionListArgs args)
               throws IOException,
                      org.apache.hadoop.yarn.exceptions.YarnException

Implement the list action.

Specified by:

actionList in interface SliderClientAPI

Parameters:

clustername - List out specific instance name

args - Action list arguments

Returns:

0 if one or more entries were listed

Throws:

IOException

org.apache.hadoop.yarn.exceptions.YarnException

UnknownApplicationInstanceException - if a specific instance was named but it was not found

getApplicationList

public Set<String> getApplicationList(String clustername)
                               throws IOException,
                                      org.apache.hadoop.yarn.exceptions.YarnException

Retrieve a list of all live instances. If clustername is supplied then it returns this specific cluster, if and only if it exists and is live.

Parameters:

clustername - cluster name (if looking for a specific live cluster)

Returns:

the list of application names which satisfies the list criteria

Throws:

IOException

org.apache.hadoop.yarn.exceptions.YarnException

getApplicationList

public Set<String> getApplicationList(String clustername,
                             ActionListArgs args)
                               throws IOException,
                                      org.apache.hadoop.yarn.exceptions.YarnException

Retrieve a list of application instances satisfying the query criteria.

Parameters:

clustername - List out specific instance name (set null for all)

args - Action list arguments

Returns:

the list of application names which satisfies the list criteria

Throws:

IOException

org.apache.hadoop.yarn.exceptions.YarnException

UnknownApplicationInstanceException - if a specific instance was named but it was not found

getContainers

public List<ContainerInformation> getContainers(String name)
                                         throws org.apache.hadoop.yarn.exceptions.YarnException,
                                                IOException

Throws:

org.apache.hadoop.yarn.exceptions.YarnException

IOException

enumSliderInstances

public Map<String,SliderInstanceDescription> enumSliderInstances(boolean listOnlyInState,
                                                        org.apache.hadoop.yarn.api.records.YarnApplicationState minAppState,
                                                        org.apache.hadoop.yarn.api.records.YarnApplicationState maxAppState)
                                                          throws IOException,
                                                                 org.apache.hadoop.yarn.exceptions.YarnException

Enumerate slider instances for the current user, and the most recent app report, where available.

Specified by:

enumSliderInstances in interface SliderClientAPI

Parameters:

listOnlyInState - boolean to indicate that the instances should only include those in a YARN state minAppState <= currentState <= maxAppState

minAppState - minimum application state to include in enumeration.

maxAppState - maximum application state to include

Returns:

a map of application instance name to description

Throws:

IOException - Any IO problem

org.apache.hadoop.yarn.exceptions.YarnException - YARN problems

isApplicationActive

public boolean isApplicationActive(org.apache.hadoop.yarn.api.records.ApplicationReport report)

Is an application active: accepted or running

Parameters:

report - the application report

Returns:

true if it is running or scheduled to run.

actionFlex

public int actionFlex(String name,
             ActionFlexArgs args)
               throws org.apache.hadoop.yarn.exceptions.YarnException,
                      IOException

Implement the islive action: probe for a cluster of the given name existing

Specified by:

actionFlex in interface SliderClientAPI

Returns:

exit code

Throws:

org.apache.hadoop.yarn.exceptions.YarnException

IOException

actionExists

public int actionExists(String name,
               boolean checkLive)
                 throws org.apache.hadoop.yarn.exceptions.YarnException,
                        IOException

Description copied from interface: SliderClientAPI

Test for a cluster existing probe for a cluster of the given name existing in the filesystem. If the live param is set, it must be a live cluster

Specified by:

actionExists in interface SliderClientAPI

Returns:

exit code

Throws:

org.apache.hadoop.yarn.exceptions.YarnException

IOException

actionExists

public int actionExists(String name,
               ActionExistsArgs args)
                 throws org.apache.hadoop.yarn.exceptions.YarnException,
                        IOException

Throws:

org.apache.hadoop.yarn.exceptions.YarnException

IOException

actionKillContainer

public int actionKillContainer(String name,
                      ActionKillContainerArgs args)
                        throws org.apache.hadoop.yarn.exceptions.YarnException,
                               IOException

Description copied from interface: SliderClientAPI

Kill a specific container of the cluster

Specified by:

actionKillContainer in interface SliderClientAPI

Parameters:

name - cluster name

args - arguments

Returns:

exit code

Throws:

org.apache.hadoop.yarn.exceptions.YarnException

IOException

actionEcho

public String actionEcho(String name,
                ActionEchoArgs args)
                  throws org.apache.hadoop.yarn.exceptions.YarnException,
                         IOException

Description copied from interface: SliderClientAPI

Echo operation (not currently wired up to command line)

Specified by:

actionEcho in interface SliderClientAPI

Parameters:

name - cluster name

args - arguments

Returns:

the echoed text

Throws:

org.apache.hadoop.yarn.exceptions.YarnException

IOException

getYarnAppListClient

public YarnAppListClient getYarnAppListClient()

Get at the service registry operations

Returns:

registry client -valid after the service is inited.

actionStatus

public int actionStatus(String clustername,
               ActionStatusArgs statusArgs)
                 throws org.apache.hadoop.yarn.exceptions.YarnException,
                        IOException

Description copied from interface: SliderClientAPI

Status operation

Specified by:

actionStatus in interface SliderClientAPI

Parameters:

clustername - cluster name

statusArgs - status arguments

Returns:

0 -for success, else an exception is thrown

Throws:

org.apache.hadoop.yarn.exceptions.YarnException

IOException

actionVersion

public int actionVersion()

Description copied from interface: SliderClientAPI

Version Details

Specified by:

actionVersion in interface SliderClientAPI

Returns:

exit code

actionFreeze

public int actionFreeze(String clustername,
               ActionFreezeArgs freezeArgs)
                 throws org.apache.hadoop.yarn.exceptions.YarnException,
                        IOException

Description copied from interface: SliderClientAPI

Stop the cluster

Specified by:

actionFreeze in interface SliderClientAPI

Parameters:

clustername - cluster name

freezeArgs - arguments to the stop

Returns:

EXIT_SUCCESS if the cluster was not running by the end of the operation

Throws:

org.apache.hadoop.yarn.exceptions.YarnException

IOException

actionThaw

public int actionThaw(String clustername,
             ActionThawArgs thaw)
               throws org.apache.hadoop.yarn.exceptions.YarnException,
                      IOException

Description copied from interface: SliderClientAPI

Restore a cluster

Specified by:

actionThaw in interface SliderClientAPI

Throws:

org.apache.hadoop.yarn.exceptions.YarnException

IOException

flex

public int flex(String clustername,
       Map<String,Integer> roleInstances)
         throws org.apache.hadoop.yarn.exceptions.YarnException,
                IOException

Implement flexing

Parameters:

clustername - name of the cluster

roleInstances - map of new role instances

Returns:

EXIT_SUCCESS if the #of nodes in a live cluster changed

Throws:

org.apache.hadoop.yarn.exceptions.YarnException

IOException

validateInstanceDefinition

protected void validateInstanceDefinition(AbstractClientProvider provider,
                              AggregateConf instanceDefinition,
                              SliderFileSystem fs)
                                   throws SliderException

Validate an instance definition against a provider.

Parameters:

provider - the provider performing the validation

instanceDefinition - the instance definition

Throws:

SliderException - if invalid.

loadPersistedClusterDescription

public AggregateConf loadPersistedClusterDescription(String clustername)
                                              throws IOException,
                                                     SliderException,
                                                     LockAcquireFailedException

Load the persistent cluster description

Parameters:

clustername - name of the cluster

Returns:

the description in the filesystem

Throws:

IOException - any problems loading -including a missing file

SliderException

LockAcquireFailedException

getClusterDescription

public ClusterDescription getClusterDescription(String clustername)
                                         throws org.apache.hadoop.yarn.exceptions.YarnException,
                                                IOException

Connect to a live cluster and get its current state

Parameters:

clustername - the cluster name

Returns:

its description

Throws:

org.apache.hadoop.yarn.exceptions.YarnException

IOException

getClusterDescription

public ClusterDescription getClusterDescription()
                                         throws org.apache.hadoop.yarn.exceptions.YarnException,
                                                IOException

Connect to the cluster and get its current state

Returns:

its description

Throws:

org.apache.hadoop.yarn.exceptions.YarnException

IOException

listNodeUUIDsByRole

public String[] listNodeUUIDsByRole(String role)
                             throws IOException,
                                    org.apache.hadoop.yarn.exceptions.YarnException

List all node UUIDs in a role

Parameters:

role - role name or "" for all

Returns:

an array of UUID strings

Throws:

IOException

org.apache.hadoop.yarn.exceptions.YarnException

listClusterNodesInRole

public List<ClusterNode> listClusterNodesInRole(String role)
                                         throws IOException,
                                                org.apache.hadoop.yarn.exceptions.YarnException

List all nodes in a role. This is a double round trip: once to list the nodes in a role, another to get their details

Parameters:

role - component/role to look for

Returns:

an array of ContainerNode instances

Throws:

IOException

org.apache.hadoop.yarn.exceptions.YarnException

listClusterNodes

public List<ClusterNode> listClusterNodes(String[] uuids)
                                   throws IOException,
                                          org.apache.hadoop.yarn.exceptions.YarnException

Get the details on a list of uuids

Parameters:

uuids - uuids to ask for

Returns:

a possibly empty list of node details

Throws:

IOException

org.apache.hadoop.yarn.exceptions.YarnException

getLiveInstanceDefinition

public AggregateConf getLiveInstanceDefinition()
                                        throws IOException,
                                               org.apache.hadoop.yarn.exceptions.YarnException

Get the instance definition from the far end

Throws:

IOException

org.apache.hadoop.yarn.exceptions.YarnException

waitForRoleInstanceLive

public int waitForRoleInstanceLive(String role,
                          long timeout)
                            throws WaitTimeoutException,
                                   IOException,
                                   org.apache.hadoop.yarn.exceptions.YarnException

Wait for an instance of a named role to be live (or past it in the lifecycle)

Parameters:

role - role to look for

timeout - time to wait

Returns:

the state. If still in CREATED, the cluster didn't come up in the time period. If LIVE, all is well. If >LIVE, it has shut for a reason

Throws:

IOException - IO

SliderException - Slider

WaitTimeoutException - if the wait timed out

org.apache.hadoop.yarn.exceptions.YarnException

unknownClusterException

public UnknownApplicationInstanceException unknownClusterException(String clustername)

Generate an exception for an unknown cluster

Parameters:

clustername - cluster name

Returns:

an exception with text and a relevant exit code

toString

public String toString()

Overrides:

toString in class org.apache.hadoop.service.AbstractService

getApplications

public List<org.apache.hadoop.yarn.api.records.ApplicationReport> getApplications()
                                                                           throws org.apache.hadoop.yarn.exceptions.YarnException,
                                                                                  IOException

Get all YARN applications

Returns:

a possibly empty list

Throws:

org.apache.hadoop.yarn.exceptions.YarnException

IOException

getApplicationReport

public org.apache.hadoop.yarn.api.records.ApplicationReport getApplicationReport(org.apache.hadoop.yarn.api.records.ApplicationId appId)
                                                                          throws org.apache.hadoop.yarn.exceptions.YarnException,
                                                                                 IOException

Throws:

org.apache.hadoop.yarn.exceptions.YarnException

IOException

getLaunchedInstanceDefinition

public AggregateConf getLaunchedInstanceDefinition()

The configuration used for deployment (after resolution). Non-null only after the client has launched the application

Returns:

the resolved configuration or null

actionResolve

public int actionResolve(ActionResolveArgs args)
                  throws org.apache.hadoop.yarn.exceptions.YarnException,
                         IOException

Description copied from interface: SliderClientAPI

Registry operation

Specified by:

actionResolve in interface SliderClientAPI

Parameters:

args - registry Arguments

Returns:

0 for success, -1 for some issues that aren't errors, just failures to retrieve information (e.g. no configurations for that entry)

Throws:

org.apache.hadoop.yarn.exceptions.YarnException - YARN problems

IOException - Network or other problems

actionRegistry

public int actionRegistry(ActionRegistryArgs registryArgs)
                   throws org.apache.hadoop.yarn.exceptions.YarnException,
                          IOException

Description copied from interface: SliderClientAPI

Registry operation

Specified by:

actionRegistry in interface SliderClientAPI

Parameters:

registryArgs - registry Arguments

Returns:

0 for success, -1 for some issues that aren't errors, just failures to retrieve information (e.g. no configurations for that entry)

Throws:

org.apache.hadoop.yarn.exceptions.YarnException - YARN problems

IOException - Network or other problems

actionRegistryList

public Collection<org.apache.hadoop.registry.client.types.ServiceRecord> actionRegistryList(ActionRegistryArgs registryArgs)
                                                                                     throws org.apache.hadoop.yarn.exceptions.YarnException,
                                                                                            IOException

Registry operation

Parameters:

registryArgs - registry Arguments

Returns:

the instances (for tests)

Throws:

org.apache.hadoop.yarn.exceptions.YarnException - YARN problems

IOException - Network or other problems

actionDiagnostic

public int actionDiagnostic(ActionDiagnosticArgs diagnosticArgs)

Description copied from interface: SliderClientAPI

diagnostic operation

Specified by:

actionDiagnostic in interface SliderClientAPI

Parameters:

diagnosticArgs - diagnostic Arguments

Returns:

0 for success, -1 for some issues that aren't errors, just failures to retrieve information (e.g. no application name specified)

actionRegistryListConfigsYarn

public void actionRegistryListConfigsYarn(ActionRegistryArgs registryArgs)
                                   throws org.apache.hadoop.yarn.exceptions.YarnException,
                                          IOException

list configs available for an instance

Parameters:

registryArgs - registry Arguments

Throws:

org.apache.hadoop.yarn.exceptions.YarnException - YARN problems

IOException - Network or other problems

actionRegistryListExports

public void actionRegistryListExports(ActionRegistryArgs registryArgs)
                               throws org.apache.hadoop.yarn.exceptions.YarnException,
                                      IOException

list exports available for an instance

Parameters:

registryArgs - registry Arguments

Throws:

org.apache.hadoop.yarn.exceptions.YarnException - YARN problems

IOException - Network or other problems

actionRegistryGetConfig

public PublishedConfiguration actionRegistryGetConfig(ActionRegistryArgs registryArgs)
                                               throws org.apache.hadoop.yarn.exceptions.YarnException,
                                                      IOException

list configs available for an instance

Parameters:

registryArgs - registry Arguments

Throws:

org.apache.hadoop.yarn.exceptions.YarnException - YARN problems

IOException - Network or other problems

FileNotFoundException - if the config is not found

actionRegistryGetExport

public PublishedExports actionRegistryGetExport(ActionRegistryArgs registryArgs)
                                         throws org.apache.hadoop.yarn.exceptions.YarnException,
                                                IOException

get a specific export group

Parameters:

registryArgs - registry Arguments

Throws:

org.apache.hadoop.yarn.exceptions.YarnException - YARN problems

IOException - Network or other problems

FileNotFoundException - if the config is not found

lookupServiceRecord

public org.apache.hadoop.registry.client.types.ServiceRecord lookupServiceRecord(String serviceType,
                                                                        String id)
                                                                          throws IOException,
                                                                                 SliderException

Look up a service record of the current user

Parameters:

serviceType - service type

id - instance ID

Returns:

instance data

Throws:

UnknownApplicationInstanceException - no path or service record at the end of the path

SliderException - other failures

IOException - IO problems or wrapped exceptions

resolve

public org.apache.hadoop.registry.client.types.ServiceRecord resolve(String path)
                                                              throws IOException,
                                                                     SliderException

Look up an instance

Parameters:

path - path

Returns:

instance data

Throws:

NotFoundException - no path/no service record at the end of the path

SliderException - other failures

IOException - IO problems or wrapped exceptions

listRegistryInstances

public Map<String,org.apache.hadoop.registry.client.types.ServiceRecord> listRegistryInstances()
                                                                                        throws IOException,
                                                                                               SliderException

List instances in the registry for the current user

Returns:

a list of slider registry instances

Throws:

IOException - Any IO problem ... including no path in the registry to slider service classes for this user

SliderException - other failures

listRegisteredSliderInstances

public List<String> listRegisteredSliderInstances()
                                           throws IOException,
                                                  org.apache.hadoop.yarn.exceptions.YarnException

List instances in the registry

Returns:

the instance IDs

Throws:

IOException

org.apache.hadoop.yarn.exceptions.YarnException

getRegistryOperations

public org.apache.hadoop.registry.client.api.RegistryOperations getRegistryOperations()
                                                                               throws SliderException,
                                                                                      IOException

Description copied from interface: SliderClientAPI

Get the registry binding. As this may start the registry, it can take time and fail

Specified by:

getRegistryOperations in interface SliderClientAPI

Returns:

the registry

Throws:

SliderException

IOException

actionLookup

public int actionLookup(ActionLookupArgs args)
                 throws IOException,
                        org.apache.hadoop.yarn.exceptions.YarnException

Implement the lookup action.

Parameters:

args - Action arguments

Returns:

0 if the entry was found

Throws:

IOException

org.apache.hadoop.yarn.exceptions.YarnException

UnknownApplicationInstanceException - if a specific instance was named but it was not found

actionDependency

public int actionDependency(ActionDependencyArgs args)
                     throws IOException,
                            org.apache.hadoop.yarn.exceptions.YarnException

Description copied from interface: SliderClientAPI

Upload all Slider AM and agent dependency libraries to HDFS, so that they do not need to be uploaded with every create call. This operation is Slider version specific. So it needs to be invoked for every single version of slider/slider-client.

Specified by:

actionDependency in interface SliderClientAPI

Throws:

IOException

SliderException

org.apache.hadoop.yarn.exceptions.YarnException

listYarnClusterNodes

public NodeInformationList listYarnClusterNodes(ActionNodesArgs args)
                                         throws org.apache.hadoop.yarn.exceptions.YarnException,
                                                IOException

List the nodes in the cluster, possibly filtering by node state or label.

Specified by:

listYarnClusterNodes in interface SliderClientAPI

Parameters:

args - argument list

Returns:

a possibly empty list of nodes in the cluster

Throws:

IOException - IO problems

org.apache.hadoop.yarn.exceptions.YarnException - YARN problems

listInstanceNodes

public NodeInformationList listInstanceNodes(String instance,
                                    ActionNodesArgs args)
                                      throws org.apache.hadoop.yarn.exceptions.YarnException,
                                             IOException

List the nodes in the cluster, possibly filtering by node state or label.

Parameters:

args - argument list

Returns:

a possibly empty list of nodes in the cluster

Throws:

IOException - IO problems

org.apache.hadoop.yarn.exceptions.YarnException - YARN problems

actionNodes

public int actionNodes(String instance,
              ActionNodesArgs args)
                throws org.apache.hadoop.yarn.exceptions.YarnException,
                       IOException

List the nodes in the cluster, possibly filtering by node state or label. Prints them to stdout unless the args names a file instead.

Parameters:

args - argument list

Throws:

IOException - IO problems

org.apache.hadoop.yarn.exceptions.YarnException - YARN problems

createIpcClient

public SliderApplicationApi createIpcClient()
                                     throws IOException,
                                            org.apache.hadoop.yarn.exceptions.YarnException

Create a new IPC client for talking to slider via what follows the REST API. Client must already be bonded to the cluster

Returns:

a new IPC client

Throws:

IOException

org.apache.hadoop.yarn.exceptions.YarnException