A Data Science Central Community
Yarn Resource manager (The Yarn service Master component)
1) Controls of the total resource capacity of the cluster
2) Whatever the container is needed in the cluster it sets the minimum container size that is controlled by yarn configuration property
àyarn.scheduler.minimum-allocation-mb 1024(This value changes based on cluster ram capacity)
Description: The minimum allocation for every container request at the RM, in MBs. Memory requests lower than this won't take effect, and the specified value will get allocated at minimum
and similarly Max container size
-->yarn.scheduler.maximum-allocation-mb 8192 (This value changes based on cluster ram capacity)
Description:The maximum allocation for every container request at the RM, in MBs. Memory requests higher than this won't take effect, and will get capped to this value
3) In same way the number of cores to assign for each job.
-->yarn.scheduler.minimum-allocation-vcores 1 (value)
Description:The minimum allocation for every container request at the RM, in terms of virtual CPU cores. Requests lower than this won't take effect, and the specified value will get allocated the minimum
The maximum allocation for every container request at the RM, in terms of virtual CPU cores. Requests higher than this won't take effect, and will get cancelled.
Node manager (The yarn service worker component)
So jobs on yarn cluster runs in individual containers which is allocated by Node Manager which in turn gets permissions from Resource Manager.
So few configuration parameters of node manager those are important in context of jobs running in the containers.
Amount of physical memory, in MB, that can be allocated for containers.
Whether physical memory limits will be enforced for containers.
Whether virtual memory limits will be enforced for containers.
Number of CPU cores that can be allocated for containers.
And jobs are monitored by AM(Application master) once the application is done execution the AM gets killed .
Lets take a example here( The value in real time changes based on cluster capacity)
For a map reduce job according to the above settings the minimum container size is 1GB as defined in (yarn.scheduler.minimum-allocation-mb) and can be increased to 8 GB on the whole given in setting yarn.nodemanager.resource.memory-mb
So if the AM has started a container for MR job then MR container size can be further controlled by
mapreduce.map.memory.mb = 8Gb(for this scenario) and
The value for mapreduce.map.memory.mb must be between Yarn.scheduler.minimum-allocation-mb and
yarn.mapreduce.map.memory.mb(the high limit for mapreduce job set by mapreduce.map.memory.mb)
So the boundary value for MR job container would be 1GB on lower end and 8gb on higher end.
Because any MR job is Hadoop mapper is a java process and each Java process has its own heap memory maximum allocation settings configured via mapred.map.child.java.opts and mapreduce.map.memory.mb is the upper memory limit that Hadoop allows to be allocated to a mapper, in megabytes.
Thus, the Hadoop and the Java settings are related. The Hadoop setting is more of a resource enforcement/controlling one and the Java is more of a resource configuration one. The Java heap settings should be smaller than the Hadoop container memory limit because we need reserve memory for Java code .
IMP: In general the java heap size should be equal to 1/3 memory of the container size.
While running any complex queries in hive the container sizes for MR can be tweaked to convert shuffle joins to Map joins . If you execute the visual Explain plan for any hive statement query, then it displays the query plan .
Based on query plan tweak the container sizes and view the Visual Explain again to see weather the job is converted from shuffle to map join.
For mapreduce.map.memory.mb If this limit is exceeded, Hadoop will kill the mapper with an error like this:
Current usage: 1GB of 1536 physical memory used; 970.1 MB of 1.0 GB virtual memory used. Killing container. (displays the container id)
If the mapper process runs out of heap memory given in mapreduce.map.java.opts in Hadoop 2+ , the mapper throws a java out of memory exceptions
Error: java.lang.RuntimeException: java.lang.OutOfMemoryError
Finally: The containers are launched on local node only when the following parameter value is true
The minimum fraction of number of disks to be healthy for the nodemanager to launch new containers. This correspond to both yarn-nodemanager.local-dirs and yarn.nodemanager.log-dirs. i.e. If there are less number of healthy local-dirs (or log-dirs) available, then new containers will not be launched on this node
In a nutshell if the total cluster ram capacity is 400gb and if the TEZ container size is 10gb, the cluster environment can run 40 jobs at a time.
Check for Hadoop Hands on Code in: