org.apache.slider.providers

Class ProviderUtils

java.lang.Object

org.apache.slider.providers.ProviderUtils

All Implemented Interfaces:

RoleKeys

public class ProviderUtils
extends Object
implements RoleKeys

this is a factoring out of methods handy for providers. It's bonded to a log at construction time

Field Summary

Fields

Modifier and Type	Field and Description
protected org.slf4j.Logger	log

Fields inherited from interface org.apache.slider.api.RoleKeys

ENV_PREFIX, GC_OPTS, JVM_HEAP, JVM_OPTS, ROLE_ACTUAL_INSTANCES, ROLE_ADDITIONAL_ARGS, ROLE_FAILED_INSTANCES, ROLE_FAILED_RECENTLY_INSTANCES, ROLE_FAILED_STARTING_INSTANCES, ROLE_NAME, ROLE_NODE_FAILED_INSTANCES, ROLE_PENDING_AA_INSTANCES, ROLE_PREEMPTED_INSTANCES, ROLE_RELEASING_INSTANCES, ROLE_REQUESTED_INSTANCES

Constructor Summary

Constructors

Constructor and Description
ProviderUtils(org.slf4j.Logger log) Create an instace

Method Summary

Methods

Modifier and Type	Method and Description
static boolean	addAgentTar(Object provider, String tarName, SliderFileSystem sliderFileSystem, org.apache.hadoop.fs.Path agentDir) Add/overwrite the agent tarball (overwritten every time application is restarted)
static void	addAllDependencyJars(Map<String,org.apache.hadoop.yarn.api.records.LocalResource> providerResources, SliderFileSystem sliderFileSystem, org.apache.hadoop.fs.Path tempPath, String libDir, String libLocalSrcDir) Loads all dependency jars from the default path
static void	addDependencyJars(Map<String,org.apache.hadoop.yarn.api.records.LocalResource> providerResources, SliderFileSystem sliderFileSystem, org.apache.hadoop.fs.Path tempPath, String libdir, String[] resources, Class[] classes) Add a set of dependencies to the provider resources being built up, by copying them from the local classpath to the remote one, then registering them
static boolean	addProviderJar(Map<String,org.apache.hadoop.yarn.api.records.LocalResource> providerResources, Object provider, String jarName, SliderFileSystem sliderFileSystem, org.apache.hadoop.fs.Path tempPath, String libdir, boolean miniClusterTestRun) Add oneself to the classpath.
String	buildPathToHomeDir(AggregateConf instanceDefinition, String bindir, String script) Build the image dir.
String	buildPathToHomeDir(String imagePath, String applicationHome, String bindir, String script)
String	buildPathToScript(AggregateConf instance, String bindir, String script) Build the image dir.
String	buildPathToScript(ConfTreeOperations internal, String bindir, String script) Build the image dir.
String	buildScriptPath(String bindir, String script, String homedir)
static String	convertToAppRelativePath(File file)
static String	convertToAppRelativePath(String path)
File	findBinScriptInExpandedArchive(File base, String bindir, String script) Find a script in an expanded archive
static String	getAdditionalArgs(Map<String,String> roleOptions) Return any additional arguments (argv) to provide when starting this role
String	getLogdir() build the log directory
int	getRoleResourceRequirement(String val, int defVal, int maxVal)
String	getUserName() get the user name
void	propagateOption(MapOperations global, String optionKey, Map<String,String> sitexml, String siteKey) Propagate an option from the cluster specification option map to the site XML map, using the site key for the name
void	propagateSiteOptions(ClusterDescription clusterSpec, Map<String,String> sitexml) copy all options beginning site.
void	propagateSiteOptions(Map<String,String> options, Map<String,String> sitexml)
void	propagateSiteOptions(Map<String,String> options, Map<String,String> sitexml, String configName)
void	propagateSiteOptions(Map<String,String> options, Map<String,String> sitexml, String configName, Map<String,String> tokenMap)
void	validateNodeCount(AggregateConf instanceDescription, String name, int min, int max)
void	validateNodeCount(String name, int count, int min, int max) Validate the node count and heap size values of a node class
static void	validatePathReferencesLocalDir(String meaning, String path)

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

log

protected final org.slf4j.Logger log

Constructor Detail

ProviderUtils

public ProviderUtils(org.slf4j.Logger log)

Create an instace

Parameters:

log - log directory to use -usually the provider

Method Detail

addProviderJar

public static boolean addProviderJar(Map<String,org.apache.hadoop.yarn.api.records.LocalResource> providerResources,
                     Object provider,
                     String jarName,
                     SliderFileSystem sliderFileSystem,
                     org.apache.hadoop.fs.Path tempPath,
                     String libdir,
                     boolean miniClusterTestRun)
                              throws IOException,
                                     SliderException

Add oneself to the classpath. This does not work on minicluster test runs where the JAR is not built up

Parameters:

providerResources - map of provider resources to add these entries to

provider - provider to add

jarName - name of the jar to use

sliderFileSystem - target filesystem

tempPath - path in the cluster FS for temp files

libdir - relative directory to place resources

miniClusterTestRun -

Returns:

true if the class was found in a JAR

Throws:

FileNotFoundException - if the JAR was not found and this is NOT a mini cluster test run

IOException - IO problems

SliderException - any Slider problem

addAgentTar

public static boolean addAgentTar(Object provider,
                  String tarName,
                  SliderFileSystem sliderFileSystem,
                  org.apache.hadoop.fs.Path agentDir)
                           throws IOException

Add/overwrite the agent tarball (overwritten every time application is restarted)

Parameters:

provider -

tarName -

sliderFileSystem -

agentDir -

Returns:

true the location could be determined and the file added

Throws:

IOException

addDependencyJars

public static void addDependencyJars(Map<String,org.apache.hadoop.yarn.api.records.LocalResource> providerResources,
                     SliderFileSystem sliderFileSystem,
                     org.apache.hadoop.fs.Path tempPath,
                     String libdir,
                     String[] resources,
                     Class[] classes)
                              throws IOException,
                                     SliderException

Add a set of dependencies to the provider resources being built up, by copying them from the local classpath to the remote one, then registering them

Parameters:

providerResources - map of provider resources to add these entries to

sliderFileSystem - target filesystem

tempPath - path in the cluster FS for temp files

libdir - relative directory to place resources

resources - list of resource names (e.g. "hbase.jar"

classes - list of classes where classes[i] refers to a class in resources[i]

Throws:

IOException - IO problems

SliderException - any Slider problem

addAllDependencyJars

public static void addAllDependencyJars(Map<String,org.apache.hadoop.yarn.api.records.LocalResource> providerResources,
                        SliderFileSystem sliderFileSystem,
                        org.apache.hadoop.fs.Path tempPath,
                        String libDir,
                        String libLocalSrcDir)
                                 throws IOException,
                                        SliderException

Loads all dependency jars from the default path

Parameters:

providerResources - map of provider resources to add these entries to

sliderFileSystem - target filesystem

tempPath - path in the cluster FS for temp files

libDir - relative directory to place resources

libLocalSrcDir - explicitly supplied local libs dir

Throws:

IOException

SliderException

getLogdir

public String getLogdir()
                 throws IOException

build the log directory

Returns:

the log dir

Throws:

IOException

validateNodeCount

public void validateNodeCount(AggregateConf instanceDescription,
                     String name,
                     int min,
                     int max)
                       throws BadCommandArgumentsException

Throws:

BadCommandArgumentsException

validateNodeCount

public void validateNodeCount(String name,
                     int count,
                     int min,
                     int max)
                       throws BadCommandArgumentsException

Validate the node count and heap size values of a node class

If max <= 0: min <= count If max > 0: min <= count <= max

Parameters:

name - node class name

count - requested node count

min - requested heap size

max - maximum value.

Throws:

BadCommandArgumentsException - if the values are out of range

propagateSiteOptions

public void propagateSiteOptions(ClusterDescription clusterSpec,
                        Map<String,String> sitexml)

copy all options beginning site. into the site.xml

Parameters:

clusterSpec - cluster specification

sitexml - map for XML file to build up

propagateSiteOptions

public void propagateSiteOptions(Map<String,String> options,
                        Map<String,String> sitexml)

propagateSiteOptions

public void propagateSiteOptions(Map<String,String> options,
                        Map<String,String> sitexml,
                        String configName)

propagateSiteOptions

public void propagateSiteOptions(Map<String,String> options,
                        Map<String,String> sitexml,
                        String configName,
                        Map<String,String> tokenMap)

propagateOption

public void propagateOption(MapOperations global,
                   String optionKey,
                   Map<String,String> sitexml,
                   String siteKey)
                     throws BadConfigException

Propagate an option from the cluster specification option map to the site XML map, using the site key for the name

Parameters:

global - global config spec

optionKey - key in the option map

sitexml - map for XML file to build up

siteKey - key to assign the value to in the site XML

Throws:

BadConfigException - if the option is missing from the cluster spec

buildPathToHomeDir

public String buildPathToHomeDir(AggregateConf instanceDefinition,
                        String bindir,
                        String script)
                          throws FileNotFoundException,
                                 BadConfigException

Build the image dir. This path is relative and only valid at the far end

Parameters:

instanceDefinition - instance definition

bindir - bin subdir

script - script in bin subdir

Returns:

the path to the script

Throws:

FileNotFoundException - if a file is not found, or it is not a directory*

BadConfigException

buildPathToHomeDir

public String buildPathToHomeDir(String imagePath,
                        String applicationHome,
                        String bindir,
                        String script)
                          throws FileNotFoundException

Throws:

FileNotFoundException

buildPathToScript

public String buildPathToScript(AggregateConf instance,
                       String bindir,
                       String script)
                         throws FileNotFoundException

Build the image dir. This path is relative and only valid at the far end

Parameters:

instance - instance options

bindir - bin subdir

script - script in bin subdir

Returns:

the path to the script

Throws:

FileNotFoundException - if a file is not found, or it is not a directory*

buildPathToScript

public String buildPathToScript(ConfTreeOperations internal,
                       String bindir,
                       String script)
                         throws FileNotFoundException

Build the image dir. This path is relative and only valid at the far end

Parameters:

internal - internal options

bindir - bin subdir

script - script in bin subdir

Returns:

the path to the script

Throws:

FileNotFoundException - if a file is not found, or it is not a directory*

buildScriptPath

public String buildScriptPath(String bindir,
                     String script,
                     String homedir)

convertToAppRelativePath

public static String convertToAppRelativePath(File file)

convertToAppRelativePath

public static String convertToAppRelativePath(String path)

validatePathReferencesLocalDir

public static void validatePathReferencesLocalDir(String meaning,
                                  String path)
                                           throws BadConfigException

Throws:

BadConfigException

getUserName

public String getUserName()
                   throws IOException

get the user name

Returns:

the user name

Throws:

IOException

findBinScriptInExpandedArchive

public File findBinScriptInExpandedArchive(File base,
                                  String bindir,
                                  String script)
                                    throws FileNotFoundException

Find a script in an expanded archive

Parameters:

base - base directory

bindir - bin subdir

script - script in bin subdir

Returns:

the path to the script

Throws:

FileNotFoundException - if a file is not found, or it is not a directory

getAdditionalArgs

public static String getAdditionalArgs(Map<String,String> roleOptions)

Return any additional arguments (argv) to provide when starting this role

Parameters:

roleOptions - The options for this role

Returns:

A non-null String which contains command line arguments for this role, or the empty string.

getRoleResourceRequirement

public int getRoleResourceRequirement(String val,
                             int defVal,
                             int maxVal)