org.apache.slider.server.appmaster.state

Class AppState

java.lang.Object

org.apache.slider.server.appmaster.state.AppState

public class AppState
extends Object

The model of all the ongoing state of a Slider AM. concurrency rules: any method which begins with build is not synchronized and intended to be used during initialization.

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static class	AppState.NodeCompletionResult This is a very small class to send a multiple result back from the completion operation
static class	AppState.NodeUpdatedOutcome Return value of the onNodesUpdated(List) call.

Field Summary

Fields

Modifier and Type	Field and Description
protected static org.slf4j.Logger	log

Constructor Summary

Constructors

Constructor and Description
AppState(AbstractClusterServices recordFactory, MetricsAndMonitoring metricsAndMonitoring) Create an instance

Method Summary

Methods

Modifier and Type	Method and Description
void	buildAppMasterNode(org.apache.hadoop.yarn.api.records.ContainerId containerId, String host, int amPort, String nodeHttpAddress) build up the special master node, which lives in the live node set but has a lifecycle bonded to the AM
void	buildInstance(AppStateBindingInfo binding)
Map<Integer,String>	buildNamingMap() Build map of role ID-> name
org.apache.hadoop.yarn.api.records.Resource	buildResourceRequirements(RoleStatus role, org.apache.hadoop.yarn.api.records.Resource capability) Build up the resource requirements for this role from the cluster specification, including substituing max allowed values if the specification asked for it.
RoleStatus	buildRole(ProviderRole providerRole) Add knowledge of a role.
List<AbstractRMOperation>	cancelOutstandingAARequests() Cancel any outstanding AA Requests, building up the list of ops to cancel, removing them from RoleHistory structures and the RoleStatus entries.
List<RoleInstance>	cloneLiveContainerInfoList() Clone the live container list.
List<RoleInstance>	cloneOwnedContainerList() Clone the list of active (==owned) containers
List<RoleStatus>	cloneRoleStatusList() Get a deep clone of the role status list.
void	containerReleaseSubmitted(org.apache.hadoop.yarn.api.records.Container container) Note that a container has been submitted for release; update internal state and mark the associated ContainerInfo released field to indicate that while it is still in the active list, it has been queued for release.
void	containerStartSubmitted(org.apache.hadoop.yarn.api.records.Container container, RoleInstance instance) Notification called just before the NM is asked to start a container
ProviderRole	createDynamicProviderRole(String name, MapOperations component) Build a dynamic provider role
Map<String,Map<String,ClusterNode>>	createRoleToClusterNodeMap() Build a map of role->nodename->node-info
List<RoleInstance>	enumLiveNodesInRole(String role) Enum all nodes by role.
List<RoleInstance>	enumNodesWithRoleId(int roleId, boolean owned) enum nodes by role ID, from either the owned or live node list
List<AbstractRMOperation>	escalateOutstandingRequests() Escalate operation as triggered by external timer.
ConfTreeOperations	getAppConfSnapshot()
ApplicationLivenessInformation	getApplicationLivenessInformation() get application liveness information
float	getApplicationProgressPercentage() Return the percentage done that Slider is to have YARN display in its Web UI
ClusterDescription	getClusterStatus() Get the current view of the cluster status.
AtomicInteger	getCompletionOfNodeNotInLiveListEvent()
AtomicInteger	getCompletionOfUnknownContainerEvent()
Map<String,ComponentInformation>	getComponentInfoSnapshot() Get a snapshot of component information.
String	getContainerDiagnosticInfo() Get diagnostics info about containers
Map<org.apache.hadoop.yarn.api.records.ContainerId,RoleInstance>	getFailedContainers()
long	getFailedCountainerCount()
org.apache.hadoop.fs.Path	getHistoryPath() Get the path used for history files
AggregateConf	getInstanceDefinition()
AggregateConf	getInstanceDefinitionSnapshot()
ConfTreeOperations	getInternalsSnapshot()
Map<org.apache.hadoop.yarn.api.records.ContainerId,RoleInstance>	getLiveContainers()
RoleInstance	getLiveInstanceByContainerID(String containerId) Lookup live instance by string value of container ID
List<RoleInstance>	getLiveInstancesByContainerIDs(Collection<String> containerIDs)
protected Map<String,Integer>	getLiveStatistics() Get the live statistics map
protected String	getLogsURLForContainer(org.apache.hadoop.yarn.api.records.Container c) Get the URL log for a container
int	getNumOwnedContainers() Get the number of active (==owned) containers
RoleInstance	getOwnedContainer(org.apache.hadoop.yarn.api.records.ContainerId id) Look up an active container: any container that the AM has, even if it is not currently running/live
RoleInstance	getOwnedInstanceByContainerID(String containerId) Lookup owned instance by string value of container ID
ConfTreeOperations	getResourcesSnapshot()
RoleHistory	getRoleHistory() Get the role history of the application
protected Map<String,ProviderRole>	getRoleMap()
Map<Integer,ProviderRole>	getRolePriorityMap()
RoleStatistics	getRoleStatistics() Get the aggregate statistics across all roles
Map<Integer,RoleStatus>	getRoleStatusMap()
long	getSnapshotTime()
long	getStartedCountainerCount()
long	getStartFailedCountainerCount()
AggregateConf	getUnresolvedInstanceDefinition()
void	incFailedCountainerCount() Increment the count
protected void	incrementRequestCount(RoleStatus role) Increment the request count of a role.
void	incStartedCountainerCount() Increment the count and return the new value
void	incStartFailedCountainerCount() Increment the count and return the new value
void	initClusterStatus()
RoleInstance	innerOnNodeManagerContainerStarted(org.apache.hadoop.yarn.api.records.ContainerId containerId) container start event handler -throwing an exception on problems
boolean	isApplicationLive()
boolean	isShortLived(RoleInstance instance) Is a role short lived by the threshold set for this application
RoleStatus	lookupRoleStatus(org.apache.hadoop.yarn.api.records.Container c) Look up the status entry of a container or raise an exception
RoleStatus	lookupRoleStatus(int key) Look up the status entry of a role or raise an exception
RoleStatus	lookupRoleStatus(String name) Look up a role in the map
void	noteAMLaunched() Note that the master node has been launched, though it isn't considered live until any forked processes are running.
void	noteAMLive() AM declares ourselves live in the cluster description.
protected long	now() Current time in milliseconds.
AppState.NodeCompletionResult	onCompletedNode(org.apache.hadoop.yarn.api.records.ContainerStatus status) handle completed node in the CD -move something from the live server list to the completed server list.
void	onContainersAllocated(List<org.apache.hadoop.yarn.api.records.Container> allocatedContainers, List<ContainerAssignment> assignments, List<AbstractRMOperation> operations) Event handler for allocated containers: builds up the lists of assignment actions (what to run where), and possibly a list of operations to perform
RoleInstance	onNodeManagerContainerStarted(org.apache.hadoop.yarn.api.records.ContainerId containerId) container start event
void	onNodeManagerContainerStartFailed(org.apache.hadoop.yarn.api.records.ContainerId containerId, Throwable thrown) update the application state after a failure to start a container.
AppState.NodeUpdatedOutcome	onNodesUpdated(List<org.apache.hadoop.yarn.api.records.NodeReport> updatedNodes) Handle node update from the RM.
ClusterDescription	refreshClusterStatus() Update the cluster description with the current application state
ClusterDescription	refreshClusterStatus(Map<String,String> providerStatus) Update the cluster description with the current application state
List<AbstractRMOperation>	releaseAllContainers() Release all containers.
List<AbstractRMOperation>	releaseContainer(org.apache.hadoop.yarn.api.records.ContainerId containerId) Releases a container based on container id
void	resetFailureCounts() Reset the "recent" failure counts of all roles
List<AbstractRMOperation>	reviewRequestAndReleaseNodes() Look at where the current node state is -and whether it should be changed
protected void	setClusterStatus(ClusterDescription clusterDesc)
void	setContainerLimits(int minMemory, int maxMemory, int minCores, int maxCores) Set the container limits -the min and max values for resource requests.
void	setInitialInstanceDefinition(AggregateConf definition) Set the instance definition -this also builds the (now obsolete) cluster specification from it.
String	toString()
List<ProviderRole>	updateResourceDefinitions(ConfTree resources) The resource configuration is updated -review and update state.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

Field Detail

log

protected static final org.slf4j.Logger log

Constructor Detail

AppState

public AppState(AbstractClusterServices recordFactory,
        MetricsAndMonitoring metricsAndMonitoring)

Create an instance

Parameters:

recordFactory - factory for YARN records

metricsAndMonitoring - metrics and monitoring services

Method Detail

getFailedCountainerCount

public long getFailedCountainerCount()

incFailedCountainerCount

public void incFailedCountainerCount()

Increment the count

getStartFailedCountainerCount

public long getStartFailedCountainerCount()

incStartedCountainerCount

public void incStartedCountainerCount()

Increment the count and return the new value

getStartedCountainerCount

public long getStartedCountainerCount()

incStartFailedCountainerCount

public void incStartFailedCountainerCount()

Increment the count and return the new value

getCompletionOfNodeNotInLiveListEvent

public AtomicInteger getCompletionOfNodeNotInLiveListEvent()

getCompletionOfUnknownContainerEvent

public AtomicInteger getCompletionOfUnknownContainerEvent()

getRoleStatusMap

public Map<Integer,RoleStatus> getRoleStatusMap()

getRoleMap

protected Map<String,ProviderRole> getRoleMap()

getRolePriorityMap

public Map<Integer,ProviderRole> getRolePriorityMap()

getFailedContainers

public Map<org.apache.hadoop.yarn.api.records.ContainerId,RoleInstance> getFailedContainers()

getLiveContainers

public Map<org.apache.hadoop.yarn.api.records.ContainerId,RoleInstance> getLiveContainers()

getClusterStatus

public ClusterDescription getClusterStatus()

Get the current view of the cluster status.

Calls to refreshClusterStatus() trigger a refresh of this field.

This is read-only to the extent that changes here do not trigger updates in the application state.

Returns:

the cluster status

setClusterStatus

protected void setClusterStatus(ClusterDescription clusterDesc)

setInitialInstanceDefinition

public void setInitialInstanceDefinition(AggregateConf definition)
                                  throws BadConfigException,
                                         IOException

Set the instance definition -this also builds the (now obsolete) cluster specification from it. Important: this is for early binding and must not be used after the build operation is complete.

Parameters:

definition - initial definition

Throws:

BadConfigException

IOException

getInstanceDefinition

public AggregateConf getInstanceDefinition()

getRoleHistory

public RoleHistory getRoleHistory()

Get the role history of the application

Returns:

the role history

getHistoryPath

public org.apache.hadoop.fs.Path getHistoryPath()

Get the path used for history files

Returns:

the directory used for history files

setContainerLimits

public void setContainerLimits(int minMemory,
                      int maxMemory,
                      int minCores,
                      int maxCores)

Set the container limits -the min and max values for resource requests. All requests must be multiples of the min values.

Parameters:

minMemory - min memory MB

maxMemory - maximum memory

minCores - min v core count

maxCores - maximum cores

getResourcesSnapshot

public ConfTreeOperations getResourcesSnapshot()

getAppConfSnapshot

public ConfTreeOperations getAppConfSnapshot()

getInternalsSnapshot

public ConfTreeOperations getInternalsSnapshot()

isApplicationLive

public boolean isApplicationLive()

getSnapshotTime

public long getSnapshotTime()

getInstanceDefinitionSnapshot

public AggregateConf getInstanceDefinitionSnapshot()

getUnresolvedInstanceDefinition

public AggregateConf getUnresolvedInstanceDefinition()

buildInstance

public void buildInstance(AppStateBindingInfo binding)
                   throws BadClusterStateException,
                          BadConfigException,
                          IOException

Throws:

BadClusterStateException

BadConfigException

IOException

initClusterStatus

public void initClusterStatus()

createDynamicProviderRole

public ProviderRole createDynamicProviderRole(String name,
                                     MapOperations component)
                                       throws BadConfigException

Build a dynamic provider role

Parameters:

name - name of role

Returns:

a new provider role

Throws:

BadConfigException - bad configuration

updateResourceDefinitions

public List<ProviderRole> updateResourceDefinitions(ConfTree resources)
                                             throws BadConfigException,
                                                    IOException

The resource configuration is updated -review and update state.

Parameters:

resources - updated resources specification

Returns:

a list of any dynamically added provider roles (purely for testing purposes)

Throws:

BadConfigException

IOException

buildRole

public RoleStatus buildRole(ProviderRole providerRole)
                     throws BadConfigException

Add knowledge of a role. This is a build-time operation that is not synchronized, and should be used while setting up the system state -before servicing requests.

Parameters:

providerRole - role to add

Returns:

the role status built up

Throws:

BadConfigException - if a role of that priority already exists

buildAppMasterNode

public void buildAppMasterNode(org.apache.hadoop.yarn.api.records.ContainerId containerId,
                      String host,
                      int amPort,
                      String nodeHttpAddress)

build up the special master node, which lives in the live node set but has a lifecycle bonded to the AM

Parameters:

containerId - the AM master

host - hostname

amPort - port

nodeHttpAddress - http address: may be null

noteAMLaunched

public void noteAMLaunched()

Note that the master node has been launched, though it isn't considered live until any forked processes are running. It is NOT registered with the role history -the container is incomplete and it will just cause confusion

noteAMLive

public void noteAMLive()

AM declares ourselves live in the cluster description. This is meant to be triggered from the callback indicating the spawned process is up and running.

lookupRoleStatus

public RoleStatus lookupRoleStatus(int key)

Look up the status entry of a role or raise an exception

Parameters:

key - role ID

Returns:

the status entry

Throws:

RuntimeException - if the role cannot be found

lookupRoleStatus

public RoleStatus lookupRoleStatus(org.apache.hadoop.yarn.api.records.Container c)

Look up the status entry of a container or raise an exception

Parameters:

c - container

Returns:

the status entry

Throws:

RuntimeException - if the role cannot be found

cloneRoleStatusList

public List<RoleStatus> cloneRoleStatusList()

Get a deep clone of the role status list. Concurrent events may mean this list (or indeed, some of the role status entries) may be inconsistent

Returns:

a snapshot of the role status entries

lookupRoleStatus

public RoleStatus lookupRoleStatus(String name)
                            throws org.apache.hadoop.yarn.exceptions.YarnRuntimeException

Look up a role in the map

Parameters:

name - role name

Returns:

the instance

Throws:

org.apache.hadoop.yarn.exceptions.YarnRuntimeException - if not found

cloneOwnedContainerList

public List<RoleInstance> cloneOwnedContainerList()

Clone the list of active (==owned) containers

Returns:

the list of role instances representing all owned containers

getNumOwnedContainers

public int getNumOwnedContainers()

Get the number of active (==owned) containers

Returns:

getOwnedContainer

public RoleInstance getOwnedContainer(org.apache.hadoop.yarn.api.records.ContainerId id)

Look up an active container: any container that the AM has, even if it is not currently running/live

cloneLiveContainerInfoList

public List<RoleInstance> cloneLiveContainerInfoList()

Clone the live container list. This is synchronized.

Returns:

a snapshot of the live node list

getLiveInstanceByContainerID

public RoleInstance getLiveInstanceByContainerID(String containerId)
                                          throws NoSuchNodeException

Lookup live instance by string value of container ID

Parameters:

containerId - container ID as a string

Returns:

the role instance for that container

Throws:

NoSuchNodeException - if it does not exist

getOwnedInstanceByContainerID

public RoleInstance getOwnedInstanceByContainerID(String containerId)
                                           throws NoSuchNodeException

Lookup owned instance by string value of container ID

Parameters:

containerId - container ID as a string

Returns:

the role instance for that container

Throws:

NoSuchNodeException - if it does not exist

getLiveInstancesByContainerIDs

public List<RoleInstance> getLiveInstancesByContainerIDs(Collection<String> containerIDs)

enumLiveNodesInRole

public List<RoleInstance> enumLiveNodesInRole(String role)

Enum all nodes by role.

Parameters:

role - role, or "" for all roles

Returns:

a list of nodes, may be empty

enumNodesWithRoleId

public List<RoleInstance> enumNodesWithRoleId(int roleId,
                                     boolean owned)

enum nodes by role ID, from either the owned or live node list

Parameters:

roleId - role the container must be in

owned - flag to indicate "use owned list" rather than the smaller "live" list

Returns:

a list of nodes, may be empty

createRoleToClusterNodeMap

public Map<String,Map<String,ClusterNode>> createRoleToClusterNodeMap()

Build a map of role->nodename->node-info

Returns:

the map of Role name to list of Cluster Nodes

containerStartSubmitted

public void containerStartSubmitted(org.apache.hadoop.yarn.api.records.Container container,
                           RoleInstance instance)

Notification called just before the NM is asked to start a container

Parameters:

container - container to start

instance - clusterNode structure

containerReleaseSubmitted

public void containerReleaseSubmitted(org.apache.hadoop.yarn.api.records.Container container)
                               throws SliderInternalStateException

Note that a container has been submitted for release; update internal state and mark the associated ContainerInfo released field to indicate that while it is still in the active list, it has been queued for release.

Parameters:

container - container

Throws:

SliderInternalStateException - if there is no container of that ID on the active list

incrementRequestCount

protected void incrementRequestCount(RoleStatus role)

Increment the request count of a role.

Also updates application state counters

Parameters:

role - role being requested.

buildResourceRequirements

public org.apache.hadoop.yarn.api.records.Resource buildResourceRequirements(RoleStatus role,
                                                                    org.apache.hadoop.yarn.api.records.Resource capability)

Build up the resource requirements for this role from the cluster specification, including substituing max allowed values if the specification asked for it.

Parameters:

role - role

capability - capability to set up. A new one may be created during normalization

onNodeManagerContainerStarted

public RoleInstance onNodeManagerContainerStarted(org.apache.hadoop.yarn.api.records.ContainerId containerId)

container start event

Parameters:

containerId - container that is to be started

Returns:

the role instance, or null if there was a problem

innerOnNodeManagerContainerStarted

public RoleInstance innerOnNodeManagerContainerStarted(org.apache.hadoop.yarn.api.records.ContainerId containerId)

container start event handler -throwing an exception on problems

Parameters:

containerId - container that is to be started

Returns:

the role instance

Throws:

RuntimeException - on problems

onNodeManagerContainerStartFailed

public void onNodeManagerContainerStartFailed(org.apache.hadoop.yarn.api.records.ContainerId containerId,
                                     Throwable thrown)

update the application state after a failure to start a container. This is perhaps where blacklisting could be most useful: failure to start a container is a sign of a more serious problem than a later exit. -relayed from NMClientAsync.CallbackHandler

Parameters:

containerId - failing container

thrown - what was thrown

onNodesUpdated

public AppState.NodeUpdatedOutcome onNodesUpdated(List<org.apache.hadoop.yarn.api.records.NodeReport> updatedNodes)

Handle node update from the RM. This syncs up the node map with the RM's view

Parameters:

updatedNodes - updated nodes

isShortLived

public boolean isShortLived(RoleInstance instance)

Is a role short lived by the threshold set for this application

Parameters:

instance - instance

Returns:

true if the instance is considered short lived

now

protected long now()

Current time in milliseconds. Made protected for the option to override it in tests.

Returns:

the current time.

onCompletedNode

public AppState.NodeCompletionResult onCompletedNode(org.apache.hadoop.yarn.api.records.ContainerStatus status)

handle completed node in the CD -move something from the live server list to the completed server list.

Parameters:

status - the node that has just completed

Returns:

NodeCompletionResult

getLogsURLForContainer

protected String getLogsURLForContainer(org.apache.hadoop.yarn.api.records.Container c)

Get the URL log for a container

Parameters:

c - container

Returns:

the URL or "" if it cannot be determined

getApplicationProgressPercentage

public float getApplicationProgressPercentage()

Return the percentage done that Slider is to have YARN display in its Web UI

Returns:

an number from 0 to 100

refreshClusterStatus

public ClusterDescription refreshClusterStatus()

Update the cluster description with the current application state

refreshClusterStatus

public ClusterDescription refreshClusterStatus(Map<String,String> providerStatus)

Update the cluster description with the current application state

Parameters:

providerStatus - status from the provider for the cluster info section

getApplicationLivenessInformation

public ApplicationLivenessInformation getApplicationLivenessInformation()

get application liveness information

Returns:

a snapshot of the current liveness information

getLiveStatistics

protected Map<String,Integer> getLiveStatistics()

Get the live statistics map

Returns:

a map of statistics values, defined in the StatusKeys keylist.

getRoleStatistics

public RoleStatistics getRoleStatistics()

Get the aggregate statistics across all roles

Returns:

role statistics

getComponentInfoSnapshot

public Map<String,ComponentInformation> getComponentInfoSnapshot()

Get a snapshot of component information.

This does not include any container list, which is more expensive to create.

Returns:

a map of current role status values.

reviewRequestAndReleaseNodes

public List<AbstractRMOperation> reviewRequestAndReleaseNodes()
                                                       throws SliderInternalStateException,
                                                              TriggerClusterTeardownException

Look at where the current node state is -and whether it should be changed

Throws:

SliderInternalStateException

TriggerClusterTeardownException

resetFailureCounts

public void resetFailureCounts()

Reset the "recent" failure counts of all roles

escalateOutstandingRequests

public List<AbstractRMOperation> escalateOutstandingRequests()

Escalate operation as triggered by external timer.

Returns:

a (usually empty) list of cancel/request operations.

cancelOutstandingAARequests

public List<AbstractRMOperation> cancelOutstandingAARequests()

Cancel any outstanding AA Requests, building up the list of ops to cancel, removing them from RoleHistory structures and the RoleStatus entries.

Returns:

a (usually empty) list of cancel/request operations.

releaseContainer

public List<AbstractRMOperation> releaseContainer(org.apache.hadoop.yarn.api.records.ContainerId containerId)
                                           throws SliderInternalStateException

Releases a container based on container id

Parameters:

containerId -

Returns:

Throws:

SliderInternalStateException

releaseAllContainers

public List<AbstractRMOperation> releaseAllContainers()

Release all containers.

Returns:

a list of operations to execute

onContainersAllocated

public void onContainersAllocated(List<org.apache.hadoop.yarn.api.records.Container> allocatedContainers,
                         List<ContainerAssignment> assignments,
                         List<AbstractRMOperation> operations)

Event handler for allocated containers: builds up the lists of assignment actions (what to run where), and possibly a list of operations to perform

Parameters:

allocatedContainers - the containers allocated

assignments - the assignments of roles to containers

operations - any allocation or release operations

getContainerDiagnosticInfo

public String getContainerDiagnosticInfo()

Get diagnostics info about containers

toString

public String toString()

Overrides:

toString in class Object

buildNamingMap

public Map<Integer,String> buildNamingMap()

Build map of role ID-> name

Returns: