org.apache.slider.providers

Class AbstractProviderService

java.lang.Object

org.apache.hadoop.service.AbstractService

org.apache.slider.server.services.workflow.WorkflowSequenceService

org.apache.slider.providers.AbstractProviderService

All Implemented Interfaces:

Closeable, AutoCloseable, org.apache.hadoop.service.Service, org.apache.hadoop.service.ServiceStateChangeListener, SliderKeys, SliderXmlConfKeys, ExitCodeProvider, ProviderCore, ProviderService, RMOperationHandlerActions, ServiceParent

Direct Known Subclasses:

AgentProviderService, SliderAMProviderService

public abstract class AbstractProviderService
extends WorkflowSequenceService
implements ProviderCore, SliderKeys, ProviderService

The base class for provider services. It lets the implementations add sequences of operations, and propagates service failures upstream

Nested Class Summary

Nested classes/interfaces inherited from interface org.apache.hadoop.service.Service

org.apache.hadoop.service.Service.STATE

Field Summary

Fields

Modifier and Type	Field and Description
protected StateAccessForProviders	amState
protected URL	amWebAPI
protected QueueAccess	queueAccess
protected AgentRestOperations	restOps
protected YarnRegistryViewForProviders	yarnRegistry

Fields inherited from interface org.apache.slider.common.SliderKeys

ADDONS_DIR, AGENT_TAR, AM_DIR_PREFIX, AM_FILTER_NAME, APP_CONTAINER_HEARTBEAT_INTERVAL_SEC, APP_CONTAINER_RELEASE_TIMEOUT, APP_DEF_DIR, APP_TYPE, APP_VERSION, APP_VERSION_UNKNOWN, CLUSTER_DIRECTORY, COMP_KEYSTORE_PASSWORD_ALIAS_DEFAULT, COMP_KEYSTORE_PASSWORD_ALIAS_KEY, COMP_KEYSTORE_PASSWORD_PROPERTY_KEY, COMP_STORES_REQUIRED_KEY, COMP_TRUSTSTORE_PASSWORD_ALIAS_DEFAULT, COMP_TRUSTSTORE_PASSWORD_ALIAS_KEY, COMP_TRUSTSTORE_PASSWORD_PROPERTY_KEY, COMPONENT_AM, CRT_FILE_NAME, CRT_PASS_FILE_NAME, CSR_FILE_NAME, DATA_DIR_NAME, DEFAULT_APP_PKG, DEFAULT_GC_OPTS, DEFAULT_JVM_HEAP, DEFAULT_YARN_MEMORY, GENERATED_CONF_DIR_NAME, GSON_JAR, HADOOP_PROXY_USER, HADOOP_USER_NAME, HDP_VERSION_PROP_NAME, HISTORY_DIR_NAME, HISTORY_FILENAME_CREATION_PATTERN, HISTORY_FILENAME_GLOB_PATTERN, HISTORY_FILENAME_MATCH_PATTERN, HISTORY_FILENAME_PREFIX, HISTORY_FILENAME_SUFFIX, INFRA_DIR_NAME, JCOMMANDER_JAR, JVM_ENABLE_ASSERTIONS, JVM_ENABLE_SYSTEM_ASSERTIONS, JVM_FORCE_IPV4, JVM_JAVA_HEADLESS, KEY_ALLOWED_PORT_RANGE, KEY_AM_ALLOWED_PORT_RANGE, KEY_FILE_NAME, KEYSTORE_FILE_NAME, KEYTAB_DIR, LOCAL_TARBALL_INSTALL_SUBDIR, LOG4J_PROP_FILENAME, LOG4J_SERVER_PROP_FILENAME, PACKAGE_DIRECTORY, PASS_LEN, PROPAGATE_RESOURCE_OPTION, PROPAGATED_CONF_DIR_NAME, PROPERTY_CONF_DIR, PROPERTY_LIB_DIR, PYTHONPATH, ROLE_AM_PRIORITY_INDEX, SECURITY_DIR, SLIDER_BASE_DIRECTORY, SLIDER_CLASSPATH_EXTRA, SLIDER_CLIENT_XML, SLIDER_DEPENDENCY_DIR, SLIDER_DEPENDENCY_DIR_PERMISSIONS, SLIDER_DEPENDENCY_HDP_PARENT_DIR, SLIDER_DEPENDENCY_LOCALIZED_DIR_LINK, SLIDER_DEPENDENCY_TAR_GZ_FILE_EXT, SLIDER_DEPENDENCY_TAR_GZ_FILE_NAME, SLIDER_JAR, SLIDER_JVM_OPTS, SLIDER_PASSPHRASE, SLIDER_SERVER_XML, SLIDER_XML, SNAPSHOT_CONF_DIR_NAME, STDERR_AM, STDOUT_AM, SUBMITTED_CONF_DIR, SYSPROP_LOG_DIR, SYSPROP_LOG4J_CONFIGURATION, TMP_DIR_PREFIX, TMP_LOGDIR_PREFIX, YARN_CONTAINER_PATH

Fields inherited from interface org.apache.slider.common.SliderXmlConfKeys

CLUSTER_DIRECTORY_PERMISSIONS, DATA_DIRECTORY_PERMISSIONS, DEFAULT_CLUSTER_DIRECTORY_PERMISSIONS, DEFAULT_DATA_DIRECTORY_PERMISSIONS, DEFAULT_YARN_QUEUE, DEFAULT_YARN_QUEUE_PRIORITY, DFS_DATANODE_KERBEROS_PRINCIPAL_KEY, DFS_NAMENODE_DELEGATION_KEY_UPDATE_INTERVAL_DEFAULT, DFS_NAMENODE_DELEGATION_KEY_UPDATE_INTERVAL_KEY, DFS_NAMENODE_DELEGATION_TOKEN_ALWAYS_USE_DEFAULT, DFS_NAMENODE_DELEGATION_TOKEN_ALWAYS_USE_KEY, DFS_NAMENODE_DELEGATION_TOKEN_MAX_LIFETIME_DEFAULT, DFS_NAMENODE_DELEGATION_TOKEN_MAX_LIFETIME_KEY, DFS_NAMENODE_DELEGATION_TOKEN_RENEW_INTERVAL_DEFAULT, DFS_NAMENODE_DELEGATION_TOKEN_RENEW_INTERVAL_KEY, DFS_NAMENODE_DU_RESERVED_KEY, DFS_NAMENODE_KERBEROS_PRINCIPAL_KEY, DFS_NAMENODE_KEYTAB_FILE_KEY, FS_DEFAULT_NAME_CLASSIC, HADOOP_HTTP_FILTER_INITIALIZERS, IPC_CLIENT_FALLBACK_TO_SIMPLE_AUTH, IPC_CLIENT_RETRY_POLICY_ENABLED_DEFAULT, IPC_CLIENT_RETRY_POLICY_SPEC_DEFAULT, KEY_AM_KEYTAB_LOCAL_PATH, KEY_AM_LAUNCH_ENV, KEY_AM_LOGIN_KEYTAB_NAME, KEY_AM_LOGIN_KEYTAB_REQUIRED, KEY_AM_RESTART_LIMIT, KEY_HDFS_KEYTAB_DIR, KEY_IPC_CLIENT_RETRY_POLICY_ENABLED, KEY_IPC_CLIENT_RETRY_POLICY_SPEC, KEY_KERBEROS_PRINCIPAL, KEY_KEYSTORE_LOCATION, KEY_KEYTAB_PRINCIPAL, KEY_PROTOCOL_ACL, KEY_PROVIDER, KEY_SECURITY_ENABLED, KEY_SLIDER_AM_DEPENDENCY_CHECKS_DISABLED, KEY_SLIDER_BASE_PATH, KEY_TEMPLATE_ORIGIN, KEY_YARN_QUEUE, KEY_YARN_QUEUE_PRIORITY, PREFIX_PROVIDER, PYTHON_EXECUTABLE_PATH, REGISTRY_PATH, REGISTRY_ZK_QUORUM, X_DEV_INSECURE_DEFAULT, X_DEV_INSECURE_REQUIRED, X_DEV_INSECURE_WS

Constructor Summary

Constructors

Modifier	Constructor and Description
protected	AbstractProviderService(String name)

Method Summary

Methods

Modifier and Type	Method and Description
void	addContainerRequest(org.apache.hadoop.yarn.client.api.AMRMClient.ContainerRequest req) Issue a container request
void	applyInitialRegistryDefinitions(URL amWebURI, URL agentOpsURI, URL agentStatusURI, org.apache.hadoop.registry.client.types.ServiceRecord serviceRecord) Prior to going live -register the initial service registry data
void	bind(StateAccessForProviders stateAccessor, QueueAccess queueAccess, List<org.apache.hadoop.yarn.api.records.Container> liveContainers)
void	bindToYarnRegistry(YarnRegistryViewForProviders yarnRegistry) Bind to the YARN registry
void	buildEndpointDetails(Map<String,String> details) Build up the endpoint details for this service
Map<String,String>	buildMonitorDetails(ClusterDescription clusterDesc) Build a map of data intended for the AM webapp that is specific about this provider.
ForkedProcessService	buildProcess(String name, Map<String,String> env, List<String> commands)
Map<String,String>	buildProviderStatus()
int	cancelContainerRequests(org.apache.hadoop.yarn.api.records.Priority priority1, org.apache.hadoop.yarn.api.records.Priority priority2, int count) Remove a container request
void	cancelSingleRequest(org.apache.hadoop.yarn.client.api.AMRMClient.ContainerRequest request) Cancel a specific request
ContainerReleaseSelector	createContainerReleaseSelector() Create the container release selector for this provider...any policy can be implemented
boolean	exec(AggregateConf instanceDefinition, File confDir, Map<String,String> env, ProviderCompleted execInProgress) override point to allow a process to start executing in this container
void	execute(List<AbstractRMOperation> operations) Execute an entire list of operations
AgentRestOperations	getAgentRestOperations() Returns the agent rest operations interface.
StateAccessForProviders	getAmState()
org.apache.hadoop.conf.Configuration	getConf()
int	getExitCode() Method to get the exit code
protected ForkedProcessService	getFPSFromParentService(ServiceParent serviceParent) Given a parent service, find the one that is a forked process
String	getHumanName() Get a human friendly name for web UIs and messages
QueueAccess	getQueueAccess()
YarnRegistryViewForProviders	getYarnRegistry()
void	initializeApplicationConfiguration(AggregateConf instanceDefinition, SliderFileSystem fileSystem) No-op implementation of this method.
boolean	isSupportedRole(String role) Scan through the roles and see if it is supported.
protected ForkedProcessService	latestProcess() Return the latest forked process service that ran
protected org.apache.hadoop.conf.Configuration	loadProviderConfigurationInformation(File confDir, String siteXMLFilename) Load a specific XML configuration file for the provider config
protected void	maybeStartCommandSequence() if we are already running, start this service
void	notifyContainerCompleted(org.apache.hadoop.yarn.api.records.ContainerId containerId) Notify the providers of container completion
protected ForkedProcessService	queueCommand(String name, Map<String,String> env, List<String> commands) Create a new forked process service with the given name, environment and command list -then add it as a child for execution in the sequence.
void	rebuildContainerDetails(List<org.apache.hadoop.yarn.api.records.Container> liveContainers, String applicationId, Map<Integer,ProviderRole> providerRoles) No-op implementation of this method.
void	releaseAssignedContainer(org.apache.hadoop.yarn.api.records.ContainerId containerId) Release an assigned container
void	setAgentRestOperations(AgentRestOperations agentRestOperations)
void	setAmState(StateAccessForProviders amState)
void	validateApplicationConfiguration(AggregateConf instance, File confDir, boolean secure) No-op implementation of this method.

Methods inherited from class org.apache.slider.server.services.workflow.WorkflowSequenceService

addService, getActiveService, getPreviousService, getServices, onServiceCompleted, serviceStart, serviceStop, setStopIfNoChildServicesAtStartup, startNextService, stateChanged, toString

Methods inherited from class org.apache.hadoop.service.AbstractService

close, getBlockers, getConfig, getFailureCause, getFailureState, getLifecycleHistory, getName, getServiceState, getStartTime, init, isInState, noteFailure, putBlocker, registerGlobalListener, registerServiceListener, removeBlocker, serviceInit, setConfig, start, stop, unregisterGlobalListener, unregisterServiceListener, waitForServiceToStop

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

Methods inherited from interface org.apache.slider.providers.ProviderService

buildContainerLaunchContext, loadProviderConfigurationInformation

Methods inherited from interface org.apache.slider.providers.ProviderCore

getName, getRoles, validateInstanceDefinition

Methods inherited from interface org.apache.hadoop.service.Service

close, getBlockers, getConfig, getFailureCause, getFailureState, getLifecycleHistory, getName, getServiceState, getStartTime, init, isInState, registerServiceListener, start, stop, unregisterServiceListener, waitForServiceToStop

Field Detail

amState

protected StateAccessForProviders amState

restOps

protected AgentRestOperations restOps

amWebAPI

protected URL amWebAPI

yarnRegistry

protected YarnRegistryViewForProviders yarnRegistry

queueAccess

protected QueueAccess queueAccess

Constructor Detail

AbstractProviderService

protected AbstractProviderService(String name)

Method Detail

getConf

public org.apache.hadoop.conf.Configuration getConf()

Specified by:

getConf in interface ProviderCore

getAmState

public StateAccessForProviders getAmState()

getQueueAccess

public QueueAccess getQueueAccess()

setAmState

public void setAmState(StateAccessForProviders amState)

getHumanName

public String getHumanName()

Description copied from interface: ProviderService

Get a human friendly name for web UIs and messages

Specified by:

getHumanName in interface ProviderService

Returns:

a name string. Default is simply the service instance name.

bind

public void bind(StateAccessForProviders stateAccessor,
        QueueAccess queueAccess,
        List<org.apache.hadoop.yarn.api.records.Container> liveContainers)

Specified by:

bind in interface ProviderService

bindToYarnRegistry

public void bindToYarnRegistry(YarnRegistryViewForProviders yarnRegistry)

Description copied from interface: ProviderService

Bind to the YARN registry

Specified by:

bindToYarnRegistry in interface ProviderService

Parameters:

yarnRegistry - YARN registry

getYarnRegistry

public YarnRegistryViewForProviders getYarnRegistry()

getAgentRestOperations

public AgentRestOperations getAgentRestOperations()

Description copied from interface: ProviderService

Returns the agent rest operations interface.

Specified by:

getAgentRestOperations in interface ProviderService

Returns:

the interface if available, null otherwise.

notifyContainerCompleted

public void notifyContainerCompleted(org.apache.hadoop.yarn.api.records.ContainerId containerId)

Description copied from interface: ProviderService

Notify the providers of container completion

Specified by:

notifyContainerCompleted in interface ProviderService

Parameters:

containerId - container that has completed

setAgentRestOperations

public void setAgentRestOperations(AgentRestOperations agentRestOperations)

loadProviderConfigurationInformation

protected org.apache.hadoop.conf.Configuration loadProviderConfigurationInformation(File confDir,
                                                                        String siteXMLFilename)
                                                                             throws BadCommandArgumentsException,
                                                                                    IOException

Load a specific XML configuration file for the provider config

Parameters:

confDir - configuration directory

siteXMLFilename - provider-specific filename

Returns:

a configuration to be included in status

Throws:

BadCommandArgumentsException - argument problems

IOException - IO problems

initializeApplicationConfiguration

public void initializeApplicationConfiguration(AggregateConf instanceDefinition,
                                      SliderFileSystem fileSystem)
                                        throws IOException,
                                               SliderException

No-op implementation of this method.

Specified by:

initializeApplicationConfiguration in interface ProviderService

Throws:

IOException

SliderException

validateApplicationConfiguration

public void validateApplicationConfiguration(AggregateConf instance,
                                    File confDir,
                                    boolean secure)
                                      throws IOException,
                                             SliderException

No-op implementation of this method. This is a validation of the application configuration on the AM. Here is where things like the existence of keytabs and other not-seen-client-side properties can be tested, before the actual process is spawned.

Specified by:

validateApplicationConfiguration in interface ProviderService

Parameters:

instance - clusterSpecification

confDir - configuration directory

secure - flag to indicate that secure mode checks must exist

Throws:

IOException - IO problemsn

SliderException - any failure

isSupportedRole

public boolean isSupportedRole(String role)

Scan through the roles and see if it is supported.

Specified by:

isSupportedRole in interface ProviderService

Parameters:

role - role to look for

Returns:

true if the role is known about -and therefore that a launcher thread can be deployed to launch it

exec

public boolean exec(AggregateConf instanceDefinition,
           File confDir,
           Map<String,String> env,
           ProviderCompleted execInProgress)
             throws IOException,
                    SliderException

override point to allow a process to start executing in this container

Specified by:

exec in interface ProviderService

Parameters:

instanceDefinition - cluster description

confDir - configuration directory

env - environment

execInProgress - the callback for the exec events

Returns:

false

Throws:

IOException

SliderException

getExitCode

public int getExitCode()

Description copied from interface: ExitCodeProvider

Method to get the exit code

Specified by:

getExitCode in interface ExitCodeProvider

Returns:

the exit code

latestProcess

protected ForkedProcessService latestProcess()

Return the latest forked process service that ran

Returns:

the forkes service

getFPSFromParentService

protected ForkedProcessService getFPSFromParentService(ServiceParent serviceParent)

Given a parent service, find the one that is a forked process

Parameters:

serviceParent - parent

Returns:

the forked process service or null if there is none

maybeStartCommandSequence

protected void maybeStartCommandSequence()

if we are already running, start this service

queueCommand

protected ForkedProcessService queueCommand(String name,
                                Map<String,String> env,
                                List<String> commands)
                                     throws IOException,
                                            SliderException

Create a new forked process service with the given name, environment and command list -then add it as a child for execution in the sequence.

Parameters:

name - command name

env - environment

commands - command line

Throws:

IOException

SliderException

buildProcess

public ForkedProcessService buildProcess(String name,
                                Map<String,String> env,
                                List<String> commands)
                                  throws IOException,
                                         SliderException

Throws:

IOException

SliderException

buildProviderStatus

public Map<String,String> buildProviderStatus()

Specified by:

buildProviderStatus in interface ProviderService

buildMonitorDetails

public Map<String,String> buildMonitorDetails(ClusterDescription clusterDesc)

Description copied from interface: ProviderService

Build a map of data intended for the AM webapp that is specific about this provider. The key is some text to be displayed, and the value can be a URL that will create an anchor over the key text. If no anchor is needed/desired, insert the key with a null value.

Specified by:

buildMonitorDetails in interface ProviderService

Returns:

the details

buildEndpointDetails

public void buildEndpointDetails(Map<String,String> details)

Description copied from interface: ProviderService

Build up the endpoint details for this service

Specified by:

buildEndpointDetails in interface ProviderService

applyInitialRegistryDefinitions

public void applyInitialRegistryDefinitions(URL amWebURI,
                                   URL agentOpsURI,
                                   URL agentStatusURI,
                                   org.apache.hadoop.registry.client.types.ServiceRecord serviceRecord)
                                     throws IOException

Description copied from interface: ProviderService

Prior to going live -register the initial service registry data

Specified by:

applyInitialRegistryDefinitions in interface ProviderService

Parameters:

amWebURI - URL to the AM. This may be proxied, so use relative paths

agentOpsURI - URI for agent operations. This will not be proxied

agentStatusURI - URI For agent status. Again: no proxy

serviceRecord - service record to build up

Throws:

IOException

createContainerReleaseSelector

public ContainerReleaseSelector createContainerReleaseSelector()

Create the container release selector for this provider...any policy can be implemented

Specified by:

createContainerReleaseSelector in interface ProviderService

Returns:

The base implementation returns the most recent containers first.

releaseAssignedContainer

public void releaseAssignedContainer(org.apache.hadoop.yarn.api.records.ContainerId containerId)

Description copied from interface: RMOperationHandlerActions

Release an assigned container

Specified by:

releaseAssignedContainer in interface RMOperationHandlerActions

Parameters:

containerId - container

addContainerRequest

public void addContainerRequest(org.apache.hadoop.yarn.client.api.AMRMClient.ContainerRequest req)

Description copied from interface: RMOperationHandlerActions

Issue a container request

Specified by:

addContainerRequest in interface RMOperationHandlerActions

cancelSingleRequest

public void cancelSingleRequest(org.apache.hadoop.yarn.client.api.AMRMClient.ContainerRequest request)

Description copied from interface: RMOperationHandlerActions

Cancel a specific request

Specified by:

cancelSingleRequest in interface RMOperationHandlerActions

Parameters:

request - request to cancel

cancelContainerRequests

public int cancelContainerRequests(org.apache.hadoop.yarn.api.records.Priority priority1,
                          org.apache.hadoop.yarn.api.records.Priority priority2,
                          int count)

Description copied from interface: RMOperationHandlerActions

Remove a container request

Specified by:

cancelContainerRequests in interface RMOperationHandlerActions

Parameters:

priority1 - priority to remove at

priority2 - second priority to target

count - number to remove

execute

public void execute(List<AbstractRMOperation> operations)

Description copied from interface: RMOperationHandlerActions

Execute an entire list of operations

Specified by:

execute in interface RMOperationHandlerActions

Parameters:

operations - ops

rebuildContainerDetails

public void rebuildContainerDetails(List<org.apache.hadoop.yarn.api.records.Container> liveContainers,
                           String applicationId,
                           Map<Integer,ProviderRole> providerRoles)

No-op implementation of this method.

Specified by:

rebuildContainerDetails in interface ProviderService