org.apache.slider.providers.agent

Class AgentClientProvider

java.lang.Object

org.apache.hadoop.conf.Configured

org.apache.slider.providers.AbstractClientProvider

org.apache.slider.providers.agent.AgentClientProvider

All Implemented Interfaces:

org.apache.hadoop.conf.Configurable, SliderKeys, SliderXmlConfKeys, AgentKeys

public class AgentClientProvider
extends AbstractClientProvider
implements AgentKeys, SliderKeys

This class implements the client-side aspects of the agent deployer

Field Summary

Fields

Modifier and Type	Field and Description
static String	E_COULD_NOT_READ_METAINFO
protected static org.slf4j.Logger	log
protected static String	NAME

Fields inherited from class org.apache.slider.providers.AbstractClientProvider

PROVIDER_RESOURCE_BASE, PROVIDER_RESOURCE_BASE_ROOT

Fields inherited from interface org.apache.slider.providers.agent.AgentKeys

ADDON_DEFINITION_DIR, ADDON_FOR_ALL_COMPONENTS, ADDON_PREFIX, ADDONS, AGENT_CONF, AGENT_CONFIG_FILE, AGENT_INSTALL_DIR, AGENT_INSTANCE_DEBUG_DATA, AGENT_MAIN_SCRIPT, AGENT_MAIN_SCRIPT_ROOT, AGENT_OUT_FILE, AGENT_VERSION, AGENT_VERSION_FILE, APP_DEF, APP_DEFINITION_DIR, APP_HOME, APP_ROOT, ARG_DEBUG, ARG_HOST, ARG_LABEL, ARG_PORT, ARG_SECURED_PORT, ARG_ZOOKEEPER_QUORUM, ARG_ZOOKEEPER_REGISTRY_PATH, CERT_FILE_LOCALIZATION_PATH, CONF_RESOURCE, CONTROLLER_URL, CREATE_DEF_ZK_NODE, HEARTBEAT_MONITOR_INTERVAL, INFRA_RUN_SECURITY_DIR, JAVA_HOME, KEY_AGENT_TWO_WAY_SSL_ENABLED, KEY_CONTAINER_LAUNCH_DELAY, PACKAGE_LIST, PACKAGE_PATH, PROVIDER_AGENT, PYTHON_EXE, ROLE_NODE, RUNAS_USER, SCRIPT_PATH, SERVICE_NAME, SYSTEM_CONFIGS, TEST_RELAX_VERIFICATION, WAIT_HEARTBEAT

Fields inherited from interface org.apache.slider.common.SliderKeys

ADDONS_DIR, AGENT_TAR, AM_DIR_PREFIX, AM_FILTER_NAME, APP_CONTAINER_HEARTBEAT_INTERVAL_SEC, APP_CONTAINER_RELEASE_TIMEOUT, APP_DEF_DIR, APP_TYPE, APP_VERSION, APP_VERSION_UNKNOWN, CLUSTER_DIRECTORY, COMP_KEYSTORE_PASSWORD_ALIAS_DEFAULT, COMP_KEYSTORE_PASSWORD_ALIAS_KEY, COMP_KEYSTORE_PASSWORD_PROPERTY_KEY, COMP_STORES_REQUIRED_KEY, COMP_TRUSTSTORE_PASSWORD_ALIAS_DEFAULT, COMP_TRUSTSTORE_PASSWORD_ALIAS_KEY, COMP_TRUSTSTORE_PASSWORD_PROPERTY_KEY, COMPONENT_AM, CRT_FILE_NAME, CRT_PASS_FILE_NAME, CSR_FILE_NAME, DATA_DIR_NAME, DEFAULT_APP_PKG, DEFAULT_GC_OPTS, DEFAULT_JVM_HEAP, DEFAULT_YARN_MEMORY, GENERATED_CONF_DIR_NAME, GSON_JAR, HADOOP_PROXY_USER, HADOOP_USER_NAME, HDP_VERSION_PROP_NAME, HISTORY_DIR_NAME, HISTORY_FILENAME_CREATION_PATTERN, HISTORY_FILENAME_GLOB_PATTERN, HISTORY_FILENAME_MATCH_PATTERN, HISTORY_FILENAME_PREFIX, HISTORY_FILENAME_SUFFIX, INFRA_DIR_NAME, JCOMMANDER_JAR, JVM_ENABLE_ASSERTIONS, JVM_ENABLE_SYSTEM_ASSERTIONS, JVM_FORCE_IPV4, JVM_JAVA_HEADLESS, KEY_ALLOWED_PORT_RANGE, KEY_AM_ALLOWED_PORT_RANGE, KEY_FILE_NAME, KEYSTORE_FILE_NAME, KEYTAB_DIR, LOCAL_TARBALL_INSTALL_SUBDIR, LOG4J_PROP_FILENAME, LOG4J_SERVER_PROP_FILENAME, PACKAGE_DIRECTORY, PASS_LEN, PROPAGATE_RESOURCE_OPTION, PROPAGATED_CONF_DIR_NAME, PROPERTY_CONF_DIR, PROPERTY_LIB_DIR, PYTHONPATH, ROLE_AM_PRIORITY_INDEX, SECURITY_DIR, SLIDER_BASE_DIRECTORY, SLIDER_CLASSPATH_EXTRA, SLIDER_CLIENT_XML, SLIDER_DEPENDENCY_DIR, SLIDER_DEPENDENCY_DIR_PERMISSIONS, SLIDER_DEPENDENCY_HDP_PARENT_DIR, SLIDER_DEPENDENCY_LOCALIZED_DIR_LINK, SLIDER_DEPENDENCY_TAR_GZ_FILE_EXT, SLIDER_DEPENDENCY_TAR_GZ_FILE_NAME, SLIDER_JAR, SLIDER_JVM_OPTS, SLIDER_PASSPHRASE, SLIDER_SERVER_XML, SLIDER_XML, SNAPSHOT_CONF_DIR_NAME, STDERR_AM, STDOUT_AM, SUBMITTED_CONF_DIR, SYSPROP_LOG_DIR, SYSPROP_LOG4J_CONFIGURATION, TMP_DIR_PREFIX, TMP_LOGDIR_PREFIX, YARN_CONTAINER_PATH

Fields inherited from interface org.apache.slider.common.SliderXmlConfKeys

CLUSTER_DIRECTORY_PERMISSIONS, DATA_DIRECTORY_PERMISSIONS, DEFAULT_CLUSTER_DIRECTORY_PERMISSIONS, DEFAULT_DATA_DIRECTORY_PERMISSIONS, DEFAULT_YARN_QUEUE, DEFAULT_YARN_QUEUE_PRIORITY, DFS_DATANODE_KERBEROS_PRINCIPAL_KEY, DFS_NAMENODE_DELEGATION_KEY_UPDATE_INTERVAL_DEFAULT, DFS_NAMENODE_DELEGATION_KEY_UPDATE_INTERVAL_KEY, DFS_NAMENODE_DELEGATION_TOKEN_ALWAYS_USE_DEFAULT, DFS_NAMENODE_DELEGATION_TOKEN_ALWAYS_USE_KEY, DFS_NAMENODE_DELEGATION_TOKEN_MAX_LIFETIME_DEFAULT, DFS_NAMENODE_DELEGATION_TOKEN_MAX_LIFETIME_KEY, DFS_NAMENODE_DELEGATION_TOKEN_RENEW_INTERVAL_DEFAULT, DFS_NAMENODE_DELEGATION_TOKEN_RENEW_INTERVAL_KEY, DFS_NAMENODE_DU_RESERVED_KEY, DFS_NAMENODE_KERBEROS_PRINCIPAL_KEY, DFS_NAMENODE_KEYTAB_FILE_KEY, FS_DEFAULT_NAME_CLASSIC, HADOOP_HTTP_FILTER_INITIALIZERS, IPC_CLIENT_FALLBACK_TO_SIMPLE_AUTH, IPC_CLIENT_RETRY_POLICY_ENABLED_DEFAULT, IPC_CLIENT_RETRY_POLICY_SPEC_DEFAULT, KEY_AM_KEYTAB_LOCAL_PATH, KEY_AM_LAUNCH_ENV, KEY_AM_LOGIN_KEYTAB_NAME, KEY_AM_LOGIN_KEYTAB_REQUIRED, KEY_AM_RESTART_LIMIT, KEY_HDFS_KEYTAB_DIR, KEY_IPC_CLIENT_RETRY_POLICY_ENABLED, KEY_IPC_CLIENT_RETRY_POLICY_SPEC, KEY_KERBEROS_PRINCIPAL, KEY_KEYSTORE_LOCATION, KEY_KEYTAB_PRINCIPAL, KEY_PROTOCOL_ACL, KEY_PROVIDER, KEY_SECURITY_ENABLED, KEY_SLIDER_AM_DEPENDENCY_CHECKS_DISABLED, KEY_SLIDER_BASE_PATH, KEY_TEMPLATE_ORIGIN, KEY_YARN_QUEUE, KEY_YARN_QUEUE_PRIORITY, PREFIX_PROVIDER, PYTHON_EXECUTABLE_PATH, REGISTRY_PATH, REGISTRY_ZK_QUORUM, X_DEV_INSECURE_DEFAULT, X_DEV_INSECURE_REQUIRED, X_DEV_INSECURE_WS

Constructor Summary

Constructors

Modifier	Constructor and Description
protected	AgentClientProvider(org.apache.hadoop.conf.Configuration conf)

Method Summary

Methods

Modifier and Type	Method and Description
Set<String>	getApplicationTags(SliderFileSystem fileSystem, String appDef) Return a set of application specific string tags.
protected org.codehaus.jettison.json.JSONObject	getCommandJson(org.codehaus.jettison.json.JSONObject defaultConfig, org.codehaus.jettison.json.JSONObject inputConfig, Metainfo metainfo, File clientInstallPath, String name)
String	getName()
List<ProviderRole>	getRoles()
void	preflightValidateClusterConfiguration(SliderFileSystem sliderFileSystem, String clustername, org.apache.hadoop.conf.Configuration configuration, AggregateConf instanceDefinition, org.apache.hadoop.fs.Path clusterDirPath, org.apache.hadoop.fs.Path generatedConfDirPath, boolean secure) This is called pre-launch to validate that the cluster specification is valid.
void	prepareAMAndConfigForLaunch(SliderFileSystem fileSystem, org.apache.hadoop.conf.Configuration serviceConf, AbstractLauncher launcher, AggregateConf instanceDefinition, org.apache.hadoop.fs.Path snapshotConfDirPath, org.apache.hadoop.fs.Path generatedConfDirPath, org.apache.hadoop.conf.Configuration clientConfExtras, String libdir, org.apache.hadoop.fs.Path tempPath, boolean miniClusterTestRun) Prepare the AM settings for launch
void	processClientOperation(SliderFileSystem fileSystem, String operation, File clientInstallPath, File appPackage, org.codehaus.jettison.json.JSONObject config, String name) Process client operations for applications such as install, configure
protected void	runCommand(File appPkgDir, File agentPkgDir, File cmdDir, String clientScript)
void	validateInstanceDefinition(AggregateConf instanceDefinition, SliderFileSystem fs) Verify that an instance definition is considered valid by the provider

Methods inherited from class org.apache.slider.providers.AbstractClientProvider

mergeTemplates, prepareInstanceConfiguration

Methods inherited from class org.apache.hadoop.conf.Configured

getConf, setConf

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

log

protected static final org.slf4j.Logger log

NAME

protected static final String NAME

See Also:

Constant Field Values

E_COULD_NOT_READ_METAINFO

public static final String E_COULD_NOT_READ_METAINFO

See Also:

Constant Field Values

Constructor Detail

AgentClientProvider

protected AgentClientProvider(org.apache.hadoop.conf.Configuration conf)

Method Detail

getName

public String getName()

Specified by:

getName in class AbstractClientProvider

getRoles

public List<ProviderRole> getRoles()

Specified by:

getRoles in class AbstractClientProvider

preflightValidateClusterConfiguration

public void preflightValidateClusterConfiguration(SliderFileSystem sliderFileSystem,
                                         String clustername,
                                         org.apache.hadoop.conf.Configuration configuration,
                                         AggregateConf instanceDefinition,
                                         org.apache.hadoop.fs.Path clusterDirPath,
                                         org.apache.hadoop.fs.Path generatedConfDirPath,
                                         boolean secure)
                                           throws SliderException,
                                                  IOException

Description copied from class: AbstractClientProvider

This is called pre-launch to validate that the cluster specification is valid. This can include checking that the security options are in the site files prior to launch, that there are no conflicting operations etc. This check is made prior to every launch of the cluster -so can pick up problems which manually edited cluster files have added, or from specification files from previous versions. The provider MUST NOT change the remote specification. This is purely a pre-launch validation of options.

Overrides:

preflightValidateClusterConfiguration in class AbstractClientProvider

Parameters:

sliderFileSystem - filesystem

clustername - name of the cluster

configuration - cluster configuration

instanceDefinition - cluster specification

clusterDirPath - directory of the cluster

generatedConfDirPath - path to place generated artifacts

secure - flag to indicate that the cluster is secure

Throws:

SliderException - on any validation issue

IOException - on any IO problem

validateInstanceDefinition

public void validateInstanceDefinition(AggregateConf instanceDefinition,
                              SliderFileSystem fs)
                                throws SliderException

Description copied from class: AbstractClientProvider

Verify that an instance definition is considered valid by the provider

Overrides:

validateInstanceDefinition in class AbstractClientProvider

Parameters:

instanceDefinition - instance definition

Throws:

SliderException - if the configuration is not valid

prepareAMAndConfigForLaunch

public void prepareAMAndConfigForLaunch(SliderFileSystem fileSystem,
                               org.apache.hadoop.conf.Configuration serviceConf,
                               AbstractLauncher launcher,
                               AggregateConf instanceDefinition,
                               org.apache.hadoop.fs.Path snapshotConfDirPath,
                               org.apache.hadoop.fs.Path generatedConfDirPath,
                               org.apache.hadoop.conf.Configuration clientConfExtras,
                               String libdir,
                               org.apache.hadoop.fs.Path tempPath,
                               boolean miniClusterTestRun)
                                 throws IOException,
                                        SliderException

Description copied from class: AbstractClientProvider

Prepare the AM settings for launch

Overrides:

prepareAMAndConfigForLaunch in class AbstractClientProvider

Parameters:

fileSystem - filesystem

serviceConf - configuration of the client

launcher - launcher to set up

instanceDefinition - instance description being launched

miniClusterTestRun - flag set to true on a mini cluster run

Throws:

IOException

SliderException

getApplicationTags

public Set<String> getApplicationTags(SliderFileSystem fileSystem,
                             String appDef)
                               throws SliderException

Description copied from class: AbstractClientProvider

Return a set of application specific string tags.

Overrides:

getApplicationTags in class AbstractClientProvider

Returns:

the set of tags.

Throws:

SliderException

processClientOperation

public void processClientOperation(SliderFileSystem fileSystem,
                          String operation,
                          File clientInstallPath,
                          File appPackage,
                          org.codehaus.jettison.json.JSONObject config,
                          String name)
                            throws SliderException

Description copied from class: AbstractClientProvider

Process client operations for applications such as install, configure

Overrides:

processClientOperation in class AbstractClientProvider

Throws:

SliderException

runCommand

protected void runCommand(File appPkgDir,
              File agentPkgDir,
              File cmdDir,
              String clientScript)
                   throws SliderException

Throws:

SliderException

getCommandJson

protected org.codehaus.jettison.json.JSONObject getCommandJson(org.codehaus.jettison.json.JSONObject defaultConfig,
                                                   org.codehaus.jettison.json.JSONObject inputConfig,
                                                   Metainfo metainfo,
                                                   File clientInstallPath,
                                                   String name)
                                                        throws SliderException

Throws:

SliderException