org.apache.slider.common.tools

Class ConfigHelper

java.lang.Object

org.apache.slider.common.tools.ConfigHelper

public class ConfigHelper
extends Object

Methods to aid in config, both in the Configuration class and with other parts of setting up Slider-initated processes. Some of the methods take an argument of a map iterable for their sources; this allows the same method

Constructor Summary

Constructors

Constructor and Description
ConfigHelper()

Method Summary

Methods

Modifier and Type	Method and Description
static void	addConfigMap(org.apache.hadoop.conf.Configuration config, Iterable<Map.Entry<String,String>> map, String origin) Set an entire map full of values
static void	addConfigMap(org.apache.hadoop.conf.Configuration config, Map<String,String> map, String origin) Set an entire map full of values
static org.apache.hadoop.conf.Configuration	addConfigurationFile(org.apache.hadoop.conf.Configuration conf, File file, boolean overwrite) Add a configuration from a file to an existing configuration
static void	addEnvironmentVariables(org.apache.hadoop.conf.Configuration conf, String prefix) Add the system env variables with the given prefix (by convention, env.)
static Map<String,String>	buildMapFromConfiguration(org.apache.hadoop.conf.Configuration conf) Take a configuration, return a hash map
static Set<String>	dumpConf(org.apache.hadoop.conf.Configuration conf) Dump the (sorted) configuration
static String	dumpConfigToString(org.apache.hadoop.conf.Configuration conf) For testing: dump a configuration
static URL	getResourceUrl(String resource) Get the URL to a resource, null if not on the CP
static boolean	injectSliderXMLResource() Inject the SliderKeys.SLIDER_XML resource into the configuration resources of all configurations.
static byte[]	loadBytes(org.apache.hadoop.fs.FileSystem fs, org.apache.hadoop.fs.Path path)
static org.apache.hadoop.conf.Configuration	loadConfFromFile(File file) Load a Hadoop configuration from a local file.
static org.apache.hadoop.conf.Configuration	loadConfFromFile(File file, boolean loadDefaults) Load a Hadoop configuration from a local file.
static org.apache.hadoop.conf.Configuration	loadConfiguration(org.apache.hadoop.fs.FileSystem fs, org.apache.hadoop.fs.Path path) Load a configuration from ANY FS path.
static org.apache.hadoop.conf.Configuration	loadFromResource(String resource) Load a configuration from a resource on this classpath.
static org.apache.hadoop.conf.Configuration	loadMandatoryResource(String resource) Load a resource that must be on the classpath
static org.apache.hadoop.conf.Configuration	loadSliderConfiguration() Load a configuration with the SliderKeys.SLIDER_XML resource included
static org.apache.hadoop.conf.Configuration	loadTemplateConfiguration(org.apache.hadoop.conf.Configuration systemConf, org.apache.hadoop.fs.Path confdir, String templateFilename, String fallbackResource) looks for the config under $confdir/$templateFilename; if not there loads it from /conf/templateFile.
static org.apache.hadoop.conf.Configuration	loadTemplateConfiguration(org.apache.hadoop.fs.FileSystem fs, org.apache.hadoop.fs.Path templatePath, String fallbackResource) looks for the config under $confdir/$templateFilename; if not there loads it from /conf/templateFile.
static org.apache.hadoop.conf.Configuration	mergeConfigurations(org.apache.hadoop.conf.Configuration base, Iterable<Map.Entry<String,String>> merge, String origin, boolean overwrite) Merge in one configuration above another
Document	parseConfiguration(org.apache.hadoop.fs.FileSystem fs, org.apache.hadoop.fs.Path path) This will load and parse a configuration to an XML document
static Document	parseConfigXML(InputStream in) Parse an XML Hadoop configuration into an XML document.
static boolean	propagate(org.apache.hadoop.conf.Configuration dest, org.apache.hadoop.conf.Configuration src, String key) Propagate a property from a source to a dest config, with a best-effort attempt at propagating the origin.
static URL	registerDefaultResource(String resource) Register a resource as a default resource.
static void	registerDeprecatedConfigItems() Register anything we consider deprecated
static org.apache.hadoop.conf.Configuration	resolveConfiguration(Iterable<Map.Entry<String,String>> keysource, org.apache.hadoop.conf.Configuration valuesource) This goes through the keyset of one configuration and retrieves each value from a value source -a different or the same configuration.
static org.apache.hadoop.fs.Path	saveConfig(org.apache.hadoop.conf.Configuration systemConf, org.apache.hadoop.conf.Configuration confToSave, org.apache.hadoop.fs.Path confdir, String filename) Save a config file in a destination directory on a given filesystem
static File	saveConfig(org.apache.hadoop.conf.Configuration generatingConf, File confdir, String filename) Generate a config file in a destination directory on the local filesystem
static void	saveConfig(org.apache.hadoop.fs.FileSystem fs, org.apache.hadoop.fs.Path destPath, org.apache.hadoop.conf.Configuration confToSave) Save a config
static Set<String>	sortedConfigKeys(Iterable<Map.Entry<String,String>> conf) Take a configuration and return a sorted set
static String	toXml(org.apache.hadoop.conf.Configuration conf) Convert to an XML string

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

ConfigHelper

public ConfigHelper()

Method Detail

dumpConf

public static Set<String> dumpConf(org.apache.hadoop.conf.Configuration conf)

Dump the (sorted) configuration

Parameters:

conf - config

Returns:

the sorted keyset

sortedConfigKeys

public static Set<String> sortedConfigKeys(Iterable<Map.Entry<String,String>> conf)

Take a configuration and return a sorted set

Parameters:

conf - config

Returns:

the sorted keyset

addConfigMap

public static void addConfigMap(org.apache.hadoop.conf.Configuration config,
                Map<String,String> map,
                String origin)
                         throws BadConfigException

Set an entire map full of values

Parameters:

config - config to patch

map - map of data

origin - origin data

Throws:

BadConfigException

addConfigMap

public static void addConfigMap(org.apache.hadoop.conf.Configuration config,
                Iterable<Map.Entry<String,String>> map,
                String origin)
                         throws BadConfigException

Set an entire map full of values

Parameters:

config - config to patch

map - map of data

origin - origin data

Throws:

BadConfigException

saveConfig

public static org.apache.hadoop.fs.Path saveConfig(org.apache.hadoop.conf.Configuration systemConf,
                                   org.apache.hadoop.conf.Configuration confToSave,
                                   org.apache.hadoop.fs.Path confdir,
                                   String filename)
                                            throws IOException

Save a config file in a destination directory on a given filesystem

Parameters:

systemConf - system conf used for creating filesystems

confToSave - config to save

confdir - the directory path where the file is to go

filename - the filename

Returns:

the destination path where the file was saved

Throws:

IOException - IO problems

saveConfig

public static void saveConfig(org.apache.hadoop.fs.FileSystem fs,
              org.apache.hadoop.fs.Path destPath,
              org.apache.hadoop.conf.Configuration confToSave)
                       throws IOException

Save a config

Parameters:

fs - filesystem

destPath - dest to save

confToSave - config to save

Throws:

IOException - IO problems

toXml

public static String toXml(org.apache.hadoop.conf.Configuration conf)
                    throws IOException

Convert to an XML string

Parameters:

conf - configuration

Returns:

conf

Throws:

IOException

parseConfiguration

public Document parseConfiguration(org.apache.hadoop.fs.FileSystem fs,
                          org.apache.hadoop.fs.Path path)
                            throws IOException

This will load and parse a configuration to an XML document

Parameters:

fs - filesystem

path - path

Returns:

an XML document

Throws:

IOException - IO failure

loadBytes

public static byte[] loadBytes(org.apache.hadoop.fs.FileSystem fs,
               org.apache.hadoop.fs.Path path)
                        throws IOException

Throws:

IOException

loadConfiguration

public static org.apache.hadoop.conf.Configuration loadConfiguration(org.apache.hadoop.fs.FileSystem fs,
                                                     org.apache.hadoop.fs.Path path)
                                                              throws IOException

Load a configuration from ANY FS path. The normal Configuration loader only works with file:// URIs

Parameters:

fs - filesystem

path - path

Returns:

a loaded resource

Throws:

IOException

saveConfig

public static File saveConfig(org.apache.hadoop.conf.Configuration generatingConf,
              File confdir,
              String filename)
                       throws IOException

Generate a config file in a destination directory on the local filesystem

Parameters:

confdir - the directory path where the file is to go

filename - the filename

Returns:

the destination path

Throws:

IOException

parseConfigXML

public static Document parseConfigXML(InputStream in)
                               throws ParserConfigurationException,
                                      IOException,
                                      SAXException

Parse an XML Hadoop configuration into an XML document. x-include is supported, but as the location isn't passed in, relative URIs are out.

Parameters:

in - instream

Returns:

a document

Throws:

ParserConfigurationException - parser feature problems

IOException - IO problems

SAXException - XML is invalid

loadConfFromFile

public static org.apache.hadoop.conf.Configuration loadConfFromFile(File file)
                                                             throws IOException

Load a Hadoop configuration from a local file.

Parameters:

file - file to load

Returns:

a configuration which hasn't actually had the load triggered yet.

Throws:

FileNotFoundException - file is not there

IOException - any other IO problem

loadConfFromFile

public static org.apache.hadoop.conf.Configuration loadConfFromFile(File file,
                                                    boolean loadDefaults)
                                                             throws IOException

Load a Hadoop configuration from a local file.

Parameters:

file - file to load

loadDefaults - flag to indicate if the defaults should be loaded yet

Returns:

a configuration which hasn't actually had the load triggered yet.

Throws:

FileNotFoundException - file is not there

IOException - any other IO problem

addConfigurationFile

public static org.apache.hadoop.conf.Configuration addConfigurationFile(org.apache.hadoop.conf.Configuration conf,
                                                        File file,
                                                        boolean overwrite)
                                                                 throws IOException

Add a configuration from a file to an existing configuration

Parameters:

conf - existing configuration

file - file to load

overwrite - flag to indicate new values should overwrite the predecessor

Returns:

the merged configuration

Throws:

IOException

addEnvironmentVariables

public static void addEnvironmentVariables(org.apache.hadoop.conf.Configuration conf,
                           String prefix)

Add the system env variables with the given prefix (by convention, env.)

Parameters:

conf - existing configuration

prefix - prefix

loadTemplateConfiguration

public static org.apache.hadoop.conf.Configuration loadTemplateConfiguration(org.apache.hadoop.conf.Configuration systemConf,
                                                             org.apache.hadoop.fs.Path confdir,
                                                             String templateFilename,
                                                             String fallbackResource)
                                                                      throws IOException

looks for the config under $confdir/$templateFilename; if not there loads it from /conf/templateFile. The property SliderXmlConfKeys.KEY_TEMPLATE_ORIGIN is set to the origin to help debug what's happening

Parameters:

systemConf - system conf

confdir - conf dir in FS

templateFilename - filename in the confdir

fallbackResource - resource to fall back on

Returns:

loaded conf

Throws:

IOException - IO problems

loadTemplateConfiguration

public static org.apache.hadoop.conf.Configuration loadTemplateConfiguration(org.apache.hadoop.fs.FileSystem fs,
                                                             org.apache.hadoop.fs.Path templatePath,
                                                             String fallbackResource)
                                                                      throws IOException

looks for the config under $confdir/$templateFilename; if not there loads it from /conf/templateFile. The property SliderXmlConfKeys.KEY_TEMPLATE_ORIGIN is set to the origin to help debug what's happening.

Parameters:

fs - Filesystem

templatePath - HDFS path for template

fallbackResource - resource to fall back on, or "" for no fallback

Returns:

loaded conf

Throws:

IOException - IO problems

FileNotFoundException - if the path doesn't have a file and there was no fallback.

dumpConfigToString

public static String dumpConfigToString(org.apache.hadoop.conf.Configuration conf)

For testing: dump a configuration

Parameters:

conf - configuration

Returns:

listing in key=value style

mergeConfigurations

public static org.apache.hadoop.conf.Configuration mergeConfigurations(org.apache.hadoop.conf.Configuration base,
                                                       Iterable<Map.Entry<String,String>> merge,
                                                       String origin,
                                                       boolean overwrite)

Merge in one configuration above another

Parameters:

base - base config

merge - one to merge. This MUST be a non-default-load config to avoid merge origin confusion

origin - description of the origin for the put operation

overwrite - flag to indicate new values should overwrite the predecessor

Returns:

the base with the merged values

registerDefaultResource

public static URL registerDefaultResource(String resource)

Register a resource as a default resource. Do not attempt to use this unless you understand that the order in which default resources are loaded affects the outcome, and that subclasses of Configuration often register new default resources

Parameters:

resource - the resource name

Returns:

the URL or null

loadFromResource

public static org.apache.hadoop.conf.Configuration loadFromResource(String resource)

Load a configuration from a resource on this classpath. If the resource is not found, an empty configuration is returned

Parameters:

resource - the resource name

Returns:

the loaded configuration.

getResourceUrl

public static URL getResourceUrl(String resource)

Get the URL to a resource, null if not on the CP

Parameters:

resource - resource to look for

Returns:

the URL or null

loadMandatoryResource

public static org.apache.hadoop.conf.Configuration loadMandatoryResource(String resource)
                                                                  throws FileNotFoundException

Load a resource that must be on the classpath

Parameters:

resource - the resource name

Returns:

the loaded configuration

Throws:

FileNotFoundException - if the resource is missing

propagate

public static boolean propagate(org.apache.hadoop.conf.Configuration dest,
                org.apache.hadoop.conf.Configuration src,
                String key)

Propagate a property from a source to a dest config, with a best-effort attempt at propagating the origin. If the

Parameters:

dest - destination

src - source

key - key to try to copy

Returns:

true if the key was found and propagated

buildMapFromConfiguration

public static Map<String,String> buildMapFromConfiguration(org.apache.hadoop.conf.Configuration conf)

Take a configuration, return a hash map

Parameters:

conf - conf

Returns:

hash map

resolveConfiguration

public static org.apache.hadoop.conf.Configuration resolveConfiguration(Iterable<Map.Entry<String,String>> keysource,
                                                        org.apache.hadoop.conf.Configuration valuesource)

This goes through the keyset of one configuration and retrieves each value from a value source -a different or the same configuration. This triggers the property resolution process of the value, resolving any variables against in-config or inherited configurations

Parameters:

keysource - source of keys

valuesource - the source of values

Returns:

a new configuration where foreach key in keysource, get(key)==valuesource.get(key)

registerDeprecatedConfigItems

public static void registerDeprecatedConfigItems()

Register anything we consider deprecated

loadSliderConfiguration

public static org.apache.hadoop.conf.Configuration loadSliderConfiguration()

Load a configuration with the SliderKeys.SLIDER_XML resource included

Returns:

a configuration instance

injectSliderXMLResource

public static boolean injectSliderXMLResource()

Inject the SliderKeys.SLIDER_XML resource into the configuration resources of all configurations.

This operation is idempotent.

If the resource is not on the classpath, downgrades, rather than fails.

Returns:

true if the resource was found and loaded.