Calculate Nums of executor and its memory in Spark

 It based  on concept of TCMR

T  - Total No. of  worker node    - Exmaple 10

C -  No. of core per Node (CPU)  -  Example 16 Core CPU 

M - Total memory (per node) GB - Example 64 GB Avaliable

R -  Reserve memory for YARN/OS (GB) Example 1 GB as reserve

Step -1 

       Reserve memory for  YARN/OS 

        example: -   1 GB

Step 2 - As per standard 4 or 5 core required for per executor as optimal solution.

             Too high is Garbage collector overhead

              Too low is Schedular overhead

Step 3 - We assume 5 core per executor as optimal solution

Step 4 -  Calculate Executor per node

                      = (CPU Core - Reserve CPU)/ core per exceutor

                      = 16-1/5

                      =15/5 =3

                      So 3 executor per node required

Step 5 - Memory of executor 

             = (Total available memory - reserve memory)/ No. Executor per node

              = (64-1)/3

              = 63/3 = 21

              =21 GB  

Step 6 -  Total no of exector on cluster

               = No. of  total nodes * no. of executor per node

               = 10 * 3 

                = 30