org.apache.slider.providers.slideram

Class SliderAMClientProvider

java.lang.Object

org.apache.hadoop.conf.Configured

org.apache.slider.providers.AbstractClientProvider

org.apache.slider.providers.slideram.SliderAMClientProvider

All Implemented Interfaces:

org.apache.hadoop.conf.Configurable, SliderKeys, SliderXmlConfKeys

public class SliderAMClientProvider
extends AbstractClientProvider
implements SliderKeys

handles the setup of the Slider AM. This keeps aspects of role, cluster validation and Clusterspec setup out of the core slider client

Field Summary

Fields

Modifier and Type	Field and Description
static String	APPCONF_JSON
static ProviderRole	APPMASTER
static String	INSTANCE_RESOURCE_BASE
static String	INTERNAL_JSON
static int	KEY_AM
protected static org.slf4j.Logger	log
protected static String	NAME
static String	RESOURCES_JSON
static List<ProviderRole>	ROLES List of roles

Fields inherited from class org.apache.slider.providers.AbstractClientProvider

PROVIDER_RESOURCE_BASE, PROVIDER_RESOURCE_BASE_ROOT, providerUtils

Fields inherited from interface org.apache.slider.common.SliderKeys

ADDONS_DIR, AGENT_TAR, AM_DIR_PREFIX, AM_FILTER_NAME, APP_CONTAINER_HEARTBEAT_INTERVAL_SEC, APP_CONTAINER_RELEASE_TIMEOUT, APP_DEF_DIR, APP_TYPE, APP_VERSION, APP_VERSION_UNKNOWN, CLUSTER_DIRECTORY, COMP_KEYSTORE_PASSWORD_ALIAS_DEFAULT, COMP_KEYSTORE_PASSWORD_ALIAS_KEY, COMP_KEYSTORE_PASSWORD_PROPERTY_KEY, COMP_STORES_REQUIRED_KEY, COMP_TRUSTSTORE_PASSWORD_ALIAS_DEFAULT, COMP_TRUSTSTORE_PASSWORD_ALIAS_KEY, COMP_TRUSTSTORE_PASSWORD_PROPERTY_KEY, COMPONENT_AM, CRT_FILE_NAME, CRT_PASS_FILE_NAME, CSR_FILE_NAME, DATA_DIR_NAME, DEFAULT_APP_PKG, DEFAULT_GC_OPTS, DEFAULT_JVM_HEAP, DEFAULT_YARN_MEMORY, GENERATED_CONF_DIR_NAME, GSON_JAR, HADOOP_PROXY_USER, HADOOP_USER_NAME, HDP_VERSION_PROP_NAME, HISTORY_DIR_NAME, HISTORY_FILENAME_CREATION_PATTERN, HISTORY_FILENAME_GLOB_PATTERN, HISTORY_FILENAME_MATCH_PATTERN, HISTORY_FILENAME_PREFIX, HISTORY_FILENAME_SUFFIX, INFRA_DIR_NAME, JCOMMANDER_JAR, JVM_ENABLE_ASSERTIONS, JVM_ENABLE_SYSTEM_ASSERTIONS, JVM_FORCE_IPV4, JVM_JAVA_HEADLESS, KEY_ALLOWED_PORT_RANGE, KEY_AM_ALLOWED_PORT_RANGE, KEY_FILE_NAME, KEYSTORE_FILE_NAME, KEYTAB_DIR, LOCAL_TARBALL_INSTALL_SUBDIR, LOG4J_PROP_FILENAME, LOG4J_SERVER_PROP_FILENAME, PACKAGE_DIRECTORY, PASS_LEN, PROPAGATE_RESOURCE_OPTION, PROPAGATED_CONF_DIR_NAME, PROPERTY_CONF_DIR, PROPERTY_LIB_DIR, PYTHONPATH, ROLE_AM_PRIORITY_INDEX, SECURITY_DIR, SLIDER_BASE_DIRECTORY, SLIDER_CLASSPATH_EXTRA, SLIDER_CLIENT_XML, SLIDER_DEPENDENCY_DIR, SLIDER_DEPENDENCY_DIR_PERMISSIONS, SLIDER_DEPENDENCY_HDP_PARENT_DIR, SLIDER_DEPENDENCY_LOCALIZED_DIR_LINK, SLIDER_DEPENDENCY_TAR_GZ_FILE_EXT, SLIDER_DEPENDENCY_TAR_GZ_FILE_NAME, SLIDER_JAR, SLIDER_JVM_OPTS, SLIDER_PASSPHRASE, SLIDER_SERVER_XML, SLIDER_XML, SNAPSHOT_CONF_DIR_NAME, STDERR_AM, STDOUT_AM, SUBMITTED_CONF_DIR, SYSPROP_LOG_DIR, SYSPROP_LOG4J_CONFIGURATION, TMP_DIR_PREFIX, TMP_LOGDIR_PREFIX, YARN_CONTAINER_PATH

Fields inherited from interface org.apache.slider.common.SliderXmlConfKeys

CLUSTER_DIRECTORY_PERMISSIONS, DATA_DIRECTORY_PERMISSIONS, DEFAULT_CLUSTER_DIRECTORY_PERMISSIONS, DEFAULT_DATA_DIRECTORY_PERMISSIONS, DEFAULT_YARN_QUEUE, DEFAULT_YARN_QUEUE_PRIORITY, DFS_DATANODE_KERBEROS_PRINCIPAL_KEY, DFS_NAMENODE_DELEGATION_KEY_UPDATE_INTERVAL_DEFAULT, DFS_NAMENODE_DELEGATION_KEY_UPDATE_INTERVAL_KEY, DFS_NAMENODE_DELEGATION_TOKEN_ALWAYS_USE_DEFAULT, DFS_NAMENODE_DELEGATION_TOKEN_ALWAYS_USE_KEY, DFS_NAMENODE_DELEGATION_TOKEN_MAX_LIFETIME_DEFAULT, DFS_NAMENODE_DELEGATION_TOKEN_MAX_LIFETIME_KEY, DFS_NAMENODE_DELEGATION_TOKEN_RENEW_INTERVAL_DEFAULT, DFS_NAMENODE_DELEGATION_TOKEN_RENEW_INTERVAL_KEY, DFS_NAMENODE_DU_RESERVED_KEY, DFS_NAMENODE_KERBEROS_PRINCIPAL_KEY, DFS_NAMENODE_KEYTAB_FILE_KEY, FS_DEFAULT_NAME_CLASSIC, HADOOP_HTTP_FILTER_INITIALIZERS, IPC_CLIENT_FALLBACK_TO_SIMPLE_AUTH, IPC_CLIENT_RETRY_POLICY_ENABLED_DEFAULT, IPC_CLIENT_RETRY_POLICY_SPEC_DEFAULT, KEY_AM_KEYTAB_LOCAL_PATH, KEY_AM_LAUNCH_ENV, KEY_AM_LOGIN_KEYTAB_NAME, KEY_AM_LOGIN_KEYTAB_REQUIRED, KEY_AM_RESTART_LIMIT, KEY_HDFS_KEYTAB_DIR, KEY_IPC_CLIENT_RETRY_POLICY_ENABLED, KEY_IPC_CLIENT_RETRY_POLICY_SPEC, KEY_KERBEROS_PRINCIPAL, KEY_KEYSTORE_LOCATION, KEY_KEYTAB_PRINCIPAL, KEY_PROTOCOL_ACL, KEY_PROVIDER, KEY_SECURITY_ENABLED, KEY_SLIDER_AM_DEPENDENCY_CHECKS_DISABLED, KEY_SLIDER_BASE_PATH, KEY_TEMPLATE_ORIGIN, KEY_YARN_QUEUE, KEY_YARN_QUEUE_PRIORITY, PREFIX_PROVIDER, PYTHON_EXECUTABLE_PATH, REGISTRY_PATH, REGISTRY_ZK_QUORUM, X_DEV_INSECURE_DEFAULT, X_DEV_INSECURE_REQUIRED, X_DEV_INSECURE_WS

Constructor Summary

Constructors

Constructor and Description
SliderAMClientProvider(org.apache.hadoop.conf.Configuration conf)

Method Summary

Methods

Modifier and Type	Method and Description
void	addJVMOptions(AggregateConf aggregateConf, JavaCommandLineBuilder cmdLine) Extract any JVM options from the cluster specification and add them to the command line
protected void	addKeytabResourceIfNecessary(SliderFileSystem fileSystem, AggregateConf instanceDescription, Map<String,org.apache.hadoop.yarn.api.records.LocalResource> providerResources) If the cluster is secure, and an HDFS installed keytab is available for AM authentication, add this keytab as a local resource for the AM launch.
String	getName()
List<ProviderRole>	getRoles()
void	preflightValidateClusterConfiguration(SliderFileSystem sliderFileSystem, String clustername, org.apache.hadoop.conf.Configuration configuration, AggregateConf instanceDefinition, org.apache.hadoop.fs.Path clusterDirPath, org.apache.hadoop.fs.Path generatedConfDirPath, boolean secure) This is called pre-launch to validate that the cluster specification is valid.
void	prepareAMAndConfigForLaunch(SliderFileSystem fileSystem, org.apache.hadoop.conf.Configuration serviceConf, AbstractLauncher launcher, AggregateConf instanceDescription, org.apache.hadoop.fs.Path snapshotConfDirPath, org.apache.hadoop.fs.Path generatedConfDirPath, org.apache.hadoop.conf.Configuration clientConfExtras, String libdir, org.apache.hadoop.fs.Path tempPath, boolean miniClusterTestRun) The Slider AM sets up all the dependency JARs above slider.jar itself Prepare the AM settings for launch
void	prepareAMResourceRequirements(MapOperations sliderAM, org.apache.hadoop.yarn.api.records.Resource capability) Update the AM resource with any local needs
void	prepareInstanceConfiguration(AggregateConf aggregateConf) Any provider-side alteration of a configuration can take place here.
void	validateInstanceDefinition(AggregateConf instanceDefinition, SliderFileSystem fs) Verify that an instance definition is considered valid by the provider

Methods inherited from class org.apache.slider.providers.AbstractClientProvider

getApplicationTags, mergeTemplates, processClientOperation

Methods inherited from class org.apache.hadoop.conf.Configured

getConf, setConf

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

log

protected static final org.slf4j.Logger log

NAME

protected static final String NAME

See Also:

Constant Field Values

INSTANCE_RESOURCE_BASE

public static final String INSTANCE_RESOURCE_BASE

See Also:

Constant Field Values

INTERNAL_JSON

public static final String INTERNAL_JSON

See Also:

Constant Field Values

APPCONF_JSON

public static final String APPCONF_JSON

See Also:

Constant Field Values

RESOURCES_JSON

public static final String RESOURCES_JSON

See Also:

Constant Field Values

ROLES

public static final List<ProviderRole> ROLES

List of roles

KEY_AM

public static final int KEY_AM

See Also:

Constant Field Values

APPMASTER

public static final ProviderRole APPMASTER

Constructor Detail

SliderAMClientProvider

public SliderAMClientProvider(org.apache.hadoop.conf.Configuration conf)

Method Detail

getName

public String getName()

Specified by:

getName in class AbstractClientProvider

getRoles

public List<ProviderRole> getRoles()

Specified by:

getRoles in class AbstractClientProvider

preflightValidateClusterConfiguration

public void preflightValidateClusterConfiguration(SliderFileSystem sliderFileSystem,
                                         String clustername,
                                         org.apache.hadoop.conf.Configuration configuration,
                                         AggregateConf instanceDefinition,
                                         org.apache.hadoop.fs.Path clusterDirPath,
                                         org.apache.hadoop.fs.Path generatedConfDirPath,
                                         boolean secure)
                                           throws SliderException,
                                                  IOException

Description copied from class: AbstractClientProvider

This is called pre-launch to validate that the cluster specification is valid. This can include checking that the security options are in the site files prior to launch, that there are no conflicting operations etc. This check is made prior to every launch of the cluster -so can pick up problems which manually edited cluster files have added, or from specification files from previous versions. The provider MUST NOT change the remote specification. This is purely a pre-launch validation of options.

Overrides:

preflightValidateClusterConfiguration in class AbstractClientProvider

Parameters:

sliderFileSystem - filesystem

clustername - name of the cluster

configuration - cluster configuration

instanceDefinition - cluster specification

clusterDirPath - directory of the cluster

generatedConfDirPath - path to place generated artifacts

secure - flag to indicate that the cluster is secure

Throws:

SliderException - on any validation issue

IOException - on any IO problem

validateInstanceDefinition

public void validateInstanceDefinition(AggregateConf instanceDefinition,
                              SliderFileSystem fs)
                                throws SliderException

Verify that an instance definition is considered valid by the provider

Overrides:

validateInstanceDefinition in class AbstractClientProvider

Parameters:

instanceDefinition - instance definition

Throws:

SliderException - if the configuration is not valid

prepareAMAndConfigForLaunch

public void prepareAMAndConfigForLaunch(SliderFileSystem fileSystem,
                               org.apache.hadoop.conf.Configuration serviceConf,
                               AbstractLauncher launcher,
                               AggregateConf instanceDescription,
                               org.apache.hadoop.fs.Path snapshotConfDirPath,
                               org.apache.hadoop.fs.Path generatedConfDirPath,
                               org.apache.hadoop.conf.Configuration clientConfExtras,
                               String libdir,
                               org.apache.hadoop.fs.Path tempPath,
                               boolean miniClusterTestRun)
                                 throws IOException,
                                        SliderException

The Slider AM sets up all the dependency JARs above slider.jar itself Prepare the AM settings for launch

Overrides:

prepareAMAndConfigForLaunch in class AbstractClientProvider

Parameters:

fileSystem - filesystem

serviceConf - configuration of the client

launcher - launcher to set up

instanceDescription - instance description being launched

miniClusterTestRun - flag set to true on a mini cluster run

Throws:

IOException

SliderException

addKeytabResourceIfNecessary

protected void addKeytabResourceIfNecessary(SliderFileSystem fileSystem,
                                AggregateConf instanceDescription,
                                Map<String,org.apache.hadoop.yarn.api.records.LocalResource> providerResources)
                                     throws IOException,
                                            BadConfigException

If the cluster is secure, and an HDFS installed keytab is available for AM authentication, add this keytab as a local resource for the AM launch.

Parameters:

fileSystem -

instanceDescription -

providerResources -

Throws:

IOException

BadConfigException - if there's no keytab and it is explicitly required.

prepareAMResourceRequirements

public void prepareAMResourceRequirements(MapOperations sliderAM,
                                 org.apache.hadoop.yarn.api.records.Resource capability)

Update the AM resource with any local needs

Parameters:

capability - capability to update

addJVMOptions

public void addJVMOptions(AggregateConf aggregateConf,
                 JavaCommandLineBuilder cmdLine)
                   throws BadConfigException

Extract any JVM options from the cluster specification and add them to the command line

Throws:

BadConfigException

prepareInstanceConfiguration

public void prepareInstanceConfiguration(AggregateConf aggregateConf)
                                  throws SliderException,
                                         IOException

Description copied from class: AbstractClientProvider

Any provider-side alteration of a configuration can take place here.

Overrides:

prepareInstanceConfiguration in class AbstractClientProvider

Parameters:

aggregateConf - config to patch

Throws:

SliderException - Slider-specific issues

IOException - IO problems