org.apache.slider.common.tools

Class CoreFileSystem

java.lang.Object

org.apache.slider.common.tools.CoreFileSystem

Direct Known Subclasses:

SliderFileSystem

public class CoreFileSystem
extends Object

Field Summary

Fields

Modifier and Type	Field and Description
protected org.apache.hadoop.conf.Configuration	configuration
protected org.apache.hadoop.fs.FileSystem	fileSystem

Constructor Summary

Constructors

Constructor and Description
CoreFileSystem(org.apache.hadoop.conf.Configuration configuration)
CoreFileSystem(org.apache.hadoop.fs.FileSystem fileSystem, org.apache.hadoop.conf.Configuration configuration)

Method Summary

Methods

Modifier and Type	Method and Description
org.apache.hadoop.fs.Path	buildAddonDirPath(String clustername, String addonId) Build up the path string for addon folder -no attempt to create the directory is made
org.apache.hadoop.fs.Path	buildAppDefDirPath(String clustername) Build up the path string for app def folder -no attempt to create the directory is made
org.apache.hadoop.fs.Path	buildClusterDirPath(String clustername) Build up the path string for a cluster instance -no attempt to create the directory is made
org.apache.hadoop.fs.Path	buildClusterSecurityDirPath(String clusterName) Build up the path string for package install location -no attempt to create the directory is made
org.apache.hadoop.fs.Path	buildKeytabInstallationDirPath(String keytabFolder) Build up the path string for keytab install location -no attempt to create the directory is made
org.apache.hadoop.fs.Path	buildKeytabPath(String keytabDir, String keytabName, String clusterName) Build up the path string for keytab install location -no attempt to create the directory is made
org.apache.hadoop.fs.Path	buildPackageDirPath(String packageName, String packageVersion) Build up the path string for package install location -no attempt to create the directory is made
void	cat(org.apache.hadoop.fs.Path path, boolean overwrite, String data)
void	copyLocalFilesToHdfs(File localPath, FilenameFilter filenameFilter, org.apache.hadoop.fs.Path destDir, org.apache.hadoop.fs.permission.FsPermission fp) Copy local file(s) to destination HDFS directory.
void	copyLocalFileToHdfs(File localPath, org.apache.hadoop.fs.Path destPath, org.apache.hadoop.fs.permission.FsPermission fp)
org.apache.hadoop.yarn.api.records.LocalResource	createAmResource(org.apache.hadoop.fs.Path destPath, org.apache.hadoop.yarn.api.records.LocalResourceType resourceType) Create an AM resource from the
org.apache.hadoop.fs.Path	createAppInstanceTempPath(String clustername, String subdir) Create the application-instance specific temporary directory in the DFS
void	createClusterDirectories(InstancePaths instancePaths) Create the Slider cluster path for a named cluster and all its subdirs This is a directory; a mkdirs() operation is executed to ensure that it is there.
org.apache.hadoop.fs.Path	createClusterDirectories(String clustername, org.apache.hadoop.conf.Configuration conf) Create the Slider cluster path for a named cluster and all its subdirs This is a directory; a mkdirs() operation is executed to ensure that it is there.
org.apache.hadoop.fs.Path	createPathThatMustExist(String uri) Create a path that must exist in the cluster fs
void	createWithPermissions(org.apache.hadoop.fs.Path dir, org.apache.hadoop.fs.permission.FsPermission clusterPerms) Create a directory with the given permissions.
org.apache.hadoop.fs.Path	getBaseApplicationPath() Get the base path
org.apache.hadoop.fs.Path	getDependencyPath() Get slider dependency parent dir in HDFS
org.apache.hadoop.fs.Path	getDependencyTarGzip() Get slider.tar.gz absolute filepath in HDFS
org.apache.hadoop.fs.FileSystem	getFileSystem() Returns the underlying FileSystem for this object.
org.apache.hadoop.fs.Path	getHomeDirectory()
org.apache.hadoop.fs.permission.FsPermission	getInstanceDirectoryPermissions()
org.apache.hadoop.fs.permission.FsPermission	getPathPermissions(org.apache.hadoop.fs.Path path) Get the permissions of a path
org.apache.hadoop.fs.Path	getTempPathForCluster(String clustername) Get the temp path for this cluster
boolean	isFile(org.apache.hadoop.fs.Path path) Given a path, check if it exists and is a file
String	listFSDir(org.apache.hadoop.fs.Path path) list entries in a filesystem directory
Map<String,org.apache.hadoop.fs.Path>	listPersistentInstances() List all application instances persisted for this user, giving the path.
org.apache.hadoop.fs.Path	locateInstanceDefinition(String clustername) Locate an application conf json in the FS.
boolean	maybeAddImagePath(Map<String,org.apache.hadoop.yarn.api.records.LocalResource> localResources, org.apache.hadoop.fs.Path imagePath)
boolean	maybeAddImagePath(Map<String,org.apache.hadoop.yarn.api.records.LocalResource> localResources, String imagePath)
org.apache.hadoop.fs.Path	purgeAppInstanceTempFiles(String clustername) Create the application-instance specific temporary directory in the DFS
Map<String,org.apache.hadoop.yarn.api.records.LocalResource>	submitDirectory(org.apache.hadoop.fs.Path srcDir, String destRelativeDir) Register all files under a fs path as a directory to push out
org.apache.hadoop.yarn.api.records.LocalResource	submitFile(File localFile, org.apache.hadoop.fs.Path tempPath, String subdir, String destFileName) Submit a local file to the filesystem references by the instance's cluster filesystem
org.apache.hadoop.yarn.api.records.LocalResource	submitJarWithClass(Class clazz, org.apache.hadoop.fs.Path tempPath, String subdir, String jarName) Submit a JAR containing a specific class, returning the resource to be mapped in
void	submitTarGzipAndUpdate(Map<String,org.apache.hadoop.yarn.api.records.LocalResource> providerResources) Submit the AM tar.gz resource referenced by the instance's cluster filesystem.
String	toString()
void	touch(org.apache.hadoop.fs.Path path, boolean overwrite)
void	verifyClusterDirectoryNonexistent(String clustername, org.apache.hadoop.fs.Path clusterDirectory) Verify that the cluster directory is not present
void	verifyClusterSpecExists(String clustername, org.apache.hadoop.fs.Path clusterSpecPath) Verify that a cluster specification exists
void	verifyDirectoryNonexistent(org.apache.hadoop.fs.Path clusterDirectory) Verify that the given directory is not present
void	verifyDirectoryWriteAccess(org.apache.hadoop.fs.Path dirPath) Verify that a user has write access to a directory.
void	verifyFileExists(org.apache.hadoop.fs.Path path) Verify that a path exists
void	verifyFileExistsInZip(org.apache.hadoop.fs.Path path, String file) Verify that a file exists in the zip file given by path
void	verifyPathExists(org.apache.hadoop.fs.Path path) Verify that a path exists

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

Field Detail

fileSystem

protected final org.apache.hadoop.fs.FileSystem fileSystem

configuration

protected final org.apache.hadoop.conf.Configuration configuration

Constructor Detail

CoreFileSystem

public CoreFileSystem(org.apache.hadoop.fs.FileSystem fileSystem,
              org.apache.hadoop.conf.Configuration configuration)

CoreFileSystem

public CoreFileSystem(org.apache.hadoop.conf.Configuration configuration)
               throws IOException

Throws:

IOException

Method Detail

getTempPathForCluster

public org.apache.hadoop.fs.Path getTempPathForCluster(String clustername)

Get the temp path for this cluster

Parameters:

clustername - name of the cluster

Returns:

path for temp files (is not purged)

getFileSystem

public org.apache.hadoop.fs.FileSystem getFileSystem()

Returns the underlying FileSystem for this object.

Returns:

filesystem

toString

public String toString()

Overrides:

toString in class Object

buildClusterDirPath

public org.apache.hadoop.fs.Path buildClusterDirPath(String clustername)

Build up the path string for a cluster instance -no attempt to create the directory is made

Parameters:

clustername - name of the cluster

Returns:

the path for persistent data

buildAppDefDirPath

public org.apache.hadoop.fs.Path buildAppDefDirPath(String clustername)

Build up the path string for app def folder -no attempt to create the directory is made

Parameters:

clustername - name of the cluster

Returns:

the path for persistent data

buildAddonDirPath

public org.apache.hadoop.fs.Path buildAddonDirPath(String clustername,
                                          String addonId)

Build up the path string for addon folder -no attempt to create the directory is made

Parameters:

clustername - name of the cluster

Returns:

the path for persistent data

buildPackageDirPath

public org.apache.hadoop.fs.Path buildPackageDirPath(String packageName,
                                            String packageVersion)

Build up the path string for package install location -no attempt to create the directory is made

Returns:

the path for persistent app package

buildClusterSecurityDirPath

public org.apache.hadoop.fs.Path buildClusterSecurityDirPath(String clusterName)

Build up the path string for package install location -no attempt to create the directory is made

Returns:

the path for persistent app package

buildKeytabInstallationDirPath

public org.apache.hadoop.fs.Path buildKeytabInstallationDirPath(String keytabFolder)

Build up the path string for keytab install location -no attempt to create the directory is made

Returns:

the path for keytab

buildKeytabPath

public org.apache.hadoop.fs.Path buildKeytabPath(String keytabDir,
                                        String keytabName,
                                        String clusterName)

Build up the path string for keytab install location -no attempt to create the directory is made

Returns:

the path for keytab installation location

createClusterDirectories

public org.apache.hadoop.fs.Path createClusterDirectories(String clustername,
                                                 org.apache.hadoop.conf.Configuration conf)
                                                   throws IOException,
                                                          SliderException

Create the Slider cluster path for a named cluster and all its subdirs This is a directory; a mkdirs() operation is executed to ensure that it is there.

Parameters:

clustername - name of the cluster

Returns:

the path to the cluster directory

Throws:

IOException - trouble

SliderException - slider-specific exceptions

createClusterDirectories

public void createClusterDirectories(InstancePaths instancePaths)
                              throws IOException,
                                     SliderException

Create the Slider cluster path for a named cluster and all its subdirs This is a directory; a mkdirs() operation is executed to ensure that it is there.

Parameters:

instancePaths - instance paths

Throws:

IOException - trouble

SliderException - slider-specific exceptions

createWithPermissions

public void createWithPermissions(org.apache.hadoop.fs.Path dir,
                         org.apache.hadoop.fs.permission.FsPermission clusterPerms)
                           throws IOException,
                                  BadClusterStateException

Create a directory with the given permissions.

Parameters:

dir - directory

clusterPerms - cluster permissions

Throws:

IOException - IO problem

BadClusterStateException - any cluster state problem

getPathPermissions

public org.apache.hadoop.fs.permission.FsPermission getPathPermissions(org.apache.hadoop.fs.Path path)
                                                                throws IOException

Get the permissions of a path

Parameters:

path - path to check

Returns:

the permissions

Throws:

IOException - any IO problem (including file not found)

getInstanceDirectoryPermissions

public org.apache.hadoop.fs.permission.FsPermission getInstanceDirectoryPermissions()

verifyClusterDirectoryNonexistent

public void verifyClusterDirectoryNonexistent(String clustername,
                                     org.apache.hadoop.fs.Path clusterDirectory)
                                       throws IOException,
                                              SliderException

Verify that the cluster directory is not present

Parameters:

clustername - name of the cluster

clusterDirectory - actual directory to look for

Throws:

IOException - trouble with FS

SliderException - If the directory exists

verifyDirectoryNonexistent

public void verifyDirectoryNonexistent(org.apache.hadoop.fs.Path clusterDirectory)
                                throws IOException,
                                       SliderException

Verify that the given directory is not present

Parameters:

clusterDirectory - actual directory to look for

Throws:

IOException - trouble with FS

SliderException - If the directory exists

verifyDirectoryWriteAccess

public void verifyDirectoryWriteAccess(org.apache.hadoop.fs.Path dirPath)
                                throws IOException,
                                       SliderException

Verify that a user has write access to a directory. It does this by creating then deleting a temp file

Parameters:

dirPath - actual directory to look for

Throws:

FileNotFoundException - file not found

IOException - trouble with FS

BadClusterStateException - if the directory is not writeable

SliderException

verifyPathExists

public void verifyPathExists(org.apache.hadoop.fs.Path path)
                      throws IOException

Verify that a path exists

Parameters:

path - path to check

Throws:

FileNotFoundException - file not found

IOException - trouble with FS

verifyFileExists

public void verifyFileExists(org.apache.hadoop.fs.Path path)
                      throws IOException

Verify that a path exists

Parameters:

path - path to check

Throws:

FileNotFoundException - file not found or is not a file

IOException - trouble with FS

isFile

public boolean isFile(org.apache.hadoop.fs.Path path)

Given a path, check if it exists and is a file

Parameters:

path - absolute path to the file to check

verifyFileExistsInZip

public void verifyFileExistsInZip(org.apache.hadoop.fs.Path path,
                         String file)
                           throws IOException

Verify that a file exists in the zip file given by path

Parameters:

path - path to zip file

file - file expected to be in zip

Throws:

FileNotFoundException - file not found or is not a zip file

IOException - trouble with FS

createAppInstanceTempPath

public org.apache.hadoop.fs.Path createAppInstanceTempPath(String clustername,
                                                  String subdir)
                                                    throws IOException

Create the application-instance specific temporary directory in the DFS

Parameters:

clustername - name of the cluster

subdir - application ID

Returns:

the path; this directory will already have been created

Throws:

IOException

purgeAppInstanceTempFiles

public org.apache.hadoop.fs.Path purgeAppInstanceTempFiles(String clustername)
                                                    throws IOException

Create the application-instance specific temporary directory in the DFS

Parameters:

clustername - name of the cluster

Returns:

the path; this directory will already have been deleted

Throws:

IOException

getBaseApplicationPath

public org.apache.hadoop.fs.Path getBaseApplicationPath()

Get the base path

Returns:

the base path optionally configured by SliderXmlConfKeys.KEY_SLIDER_BASE_PATH

getDependencyPath

public org.apache.hadoop.fs.Path getDependencyPath()

Get slider dependency parent dir in HDFS

Returns:

the parent dir path of slider.tar.gz in HDFS

getDependencyTarGzip

public org.apache.hadoop.fs.Path getDependencyTarGzip()

Get slider.tar.gz absolute filepath in HDFS

Returns:

the absolute path to slider.tar.gz in HDFS

getHomeDirectory

public org.apache.hadoop.fs.Path getHomeDirectory()

maybeAddImagePath

public boolean maybeAddImagePath(Map<String,org.apache.hadoop.yarn.api.records.LocalResource> localResources,
                        org.apache.hadoop.fs.Path imagePath)
                          throws IOException

Throws:

IOException

maybeAddImagePath

public boolean maybeAddImagePath(Map<String,org.apache.hadoop.yarn.api.records.LocalResource> localResources,
                        String imagePath)
                          throws IOException

Throws:

IOException

createAmResource

public org.apache.hadoop.yarn.api.records.LocalResource createAmResource(org.apache.hadoop.fs.Path destPath,
                                                                org.apache.hadoop.yarn.api.records.LocalResourceType resourceType)
                                                                  throws IOException

Create an AM resource from the

Parameters:

destPath - dest path in filesystem

resourceType - resource type

Returns:

the resource set up wih application-level visibility and the timestamp & size set from the file stats.

Throws:

IOException

submitDirectory

public Map<String,org.apache.hadoop.yarn.api.records.LocalResource> submitDirectory(org.apache.hadoop.fs.Path srcDir,
                                                                           String destRelativeDir)
                                                                             throws IOException

Register all files under a fs path as a directory to push out

Parameters:

srcDir - src dir

destRelativeDir - dest dir (no trailing /)

Returns:

the map of entries

Throws:

IOException

submitJarWithClass

public org.apache.hadoop.yarn.api.records.LocalResource submitJarWithClass(Class clazz,
                                                                  org.apache.hadoop.fs.Path tempPath,
                                                                  String subdir,
                                                                  String jarName)
                                                                    throws IOException,
                                                                           SliderException

Submit a JAR containing a specific class, returning the resource to be mapped in

Parameters:

clazz - class to look for

subdir - subdirectory (expected to end in a "/")

jarName - At the destination

Returns:

the local resource ref

Throws:

IOException - trouble copying to HDFS

SliderException

submitFile

public org.apache.hadoop.yarn.api.records.LocalResource submitFile(File localFile,
                                                          org.apache.hadoop.fs.Path tempPath,
                                                          String subdir,
                                                          String destFileName)
                                                            throws IOException

Submit a local file to the filesystem references by the instance's cluster filesystem

Parameters:

localFile - filename

subdir - subdirectory (expected to end in a "/")

destFileName - destination filename

Returns:

the local resource ref

Throws:

IOException - trouble copying to HDFS

submitTarGzipAndUpdate

public void submitTarGzipAndUpdate(Map<String,org.apache.hadoop.yarn.api.records.LocalResource> providerResources)
                            throws IOException,
                                   BadClusterStateException

Submit the AM tar.gz resource referenced by the instance's cluster filesystem. Also, update the providerResources object with the new resource.

Parameters:

providerResources - the provider resource map to be updated

Throws:

IOException - trouble copying to HDFS

BadClusterStateException

copyLocalFilesToHdfs

public void copyLocalFilesToHdfs(File localPath,
                        FilenameFilter filenameFilter,
                        org.apache.hadoop.fs.Path destDir,
                        org.apache.hadoop.fs.permission.FsPermission fp)
                          throws IOException

Copy local file(s) to destination HDFS directory. If localPath is a local directory then all files matching the filenameFilter (optional) are copied, otherwise filenameFilter is ignored.

Parameters:

localPath - a local file or directory path

filenameFilter - if localPath is a directory then filenameFilter is used as a filter (if specified)

destDir - the destination HDFS directory where the file(s) should be copied

fp - file permissions of all the directories and files that will be created in this api

Throws:

IOException

copyLocalFileToHdfs

public void copyLocalFileToHdfs(File localPath,
                       org.apache.hadoop.fs.Path destPath,
                       org.apache.hadoop.fs.permission.FsPermission fp)
                         throws IOException

Throws:

IOException

listFSDir

public String listFSDir(org.apache.hadoop.fs.Path path)
                 throws IOException

list entries in a filesystem directory

Parameters:

path - directory

Returns:

a listing, one to a line

Throws:

IOException

listPersistentInstances

public Map<String,org.apache.hadoop.fs.Path> listPersistentInstances()
                                                              throws IOException

List all application instances persisted for this user, giving the path. The instance name is the last element in the path

Returns:

a possibly empty map of application instance names to paths

Throws:

IOException

touch

public void touch(org.apache.hadoop.fs.Path path,
         boolean overwrite)
           throws IOException

Throws:

IOException

cat

public void cat(org.apache.hadoop.fs.Path path,
       boolean overwrite,
       String data)
         throws IOException

Throws:

IOException

createPathThatMustExist

public org.apache.hadoop.fs.Path createPathThatMustExist(String uri)
                                                  throws SliderException,
                                                         IOException

Create a path that must exist in the cluster fs

Parameters:

uri - uri to create

Returns:

the path

Throws:

SliderException - if the path does not exist

IOException

locateInstanceDefinition

public org.apache.hadoop.fs.Path locateInstanceDefinition(String clustername)
                                                   throws IOException,
                                                          SliderException

Locate an application conf json in the FS. This includes a check to verify that the file is there.

Parameters:

clustername - name of the cluster

Returns:

the path to the spec.

Throws:

IOException - IO problems

SliderException - if the path isn't there

verifyClusterSpecExists

public void verifyClusterSpecExists(String clustername,
                           org.apache.hadoop.fs.Path clusterSpecPath)
                             throws IOException,
                                    SliderException

Verify that a cluster specification exists

Parameters:

clustername - name of the cluster (For errors only)

clusterSpecPath - cluster specification path

Throws:

IOException - IO problems

SliderException - if the cluster specification is not present