Comparison of Multi Criteria Decision Making Algorithms for Ranking Cloud Renderfarm Services

Cloud services that provide a complete environment for the animators to render their files using the resources in the cloud are called Cloud Renderfarm Services. The objective of this work is to rank and compare the performance of these services using two popular Multi Criteria Decision Making (MCDM) Algorithms namely the Analytical Hierarchical Processing (AHP) and SAW (Simple Additive Weighting) methods. The performance of three real time cloud renderfarm services are ranked and compared based on five Quality of Service (QoS) attributes that are important to these services namely the Render Node Cost, File Upload Time, Availability, Elasticity and Service Response Time. The performance of these cloud renderfarm services are ranked in four different simulations by varying the weights assigned for each QoS attribute and the ranking obtained are compared. The results show that AHP and SAW assigned similar ranks to all three cloud renderfarm services for all simulations.


Introduction
Rendering is an inevitable time consuming process in the animation industry that is usually completed using a cluster of computers in the animation studio called an Offline render farm.An Offline renderfarm is a collection of render nodes.A render node is nothing but the individual computer in the cluster in which the animation scene file that is split into individual frames is rendered in a distributed manner at the same time.In order to reduce the rendering time, new technologies like grid [1][2][3][4][5] and cloud computing were also explored [6][7][8][9][10] and found to be fruitful.Among cloud services, the PaaS type of cloud renderfarm services model is gaining popularity.The advantage of using the PaaS type of cloud renderfarm services is that the animators need no prior technical knowledge about the cloud environment and they need to pay only the render node cost for every hour of rendering.To render the files in cloud based renderfarms, the animators upload the animation files to be rendered onto the service provider server.The service Provider uses software called the Rendering Job Manager (RJM), to perform the task of a queue manager and assign virtual machines in cloud based on the scheduling policy for completing the rendering tasks.The RJM scales up or scales down the number of virtual machine instantly in order to achieve the deadline.
In this work, we compare the performance of three cloud renderfarm services using two Multi Criteria Decision Making (MCDM) Algorithms namely the Analytical Hierarchical Processing (AHP) 11,12 and SAW (Simple Additive Weighting) 13 and draw useful insights.There are many works focusing on cloud services [14][15][16][17][18][19][20][21][22][23] and the ranking of Infrastructure-as-a-Service (IaaS) cloud services 24 .But very few works are aboutPaaS cloud renderfarm services 25 according to our literature survey.This work bridges this gap and contributes in the following ways: a) Identifies QoS attributes specific to Cloud Renderfarm services and ranks them using AHP and SAW (Section 2).b) Compares performance of the two MCDM algorithms based on four simulation results.C) Analyses the results obtained and provide useful insights (Section 4) and finally, conclude the work with scope for further work (Section 5).

Ranking Services Using AHP and SAW
The AHP method of ranking helps the user to specify his overall goal in the form of a hierarchical diagram in order to compare the decision alternatives efficiently.The AHP method first decomposes the Renderfarm selection problem into various levels like the overall goal to be achieved, the important criteria, sub-criteria to be considered and the decision alternatives.The overall goal in this work is to rank and select the best cloud render farm service that satisfies the user Quality of Service (QoS) requirements.The five QoS attributes were selected based on Service Measurement Index (SMI metrics)suggested by Cloud Service Measurement Index Consortium for cloud services (CSMIC).The five criteria selected for ranking the cloud Render farm services are the Render node cost, File Upload Time, Availability, Elasticity and the Service Response Time (SRT).A hierarchy diagram of QoS attributes for cloud renderfarm services is designed as given Figure 1.Next a pair wise comparison and prioritization of the attributes is performed from the lower level to the top level by estimating the relative importance of attributes considered.The relative importance of an attribute can be calculated by assigning relative weights to each attribute within each level in such a way that the sum of all the relative weights of all the attributes in each level is equal to one.The next step in this process is to compute the relative ranking for all the Sub-level attributes for which a Relative Ranking Matrix of size NR x NR is formed.Where, NR is the total number of Services to be compared for ranking.Using this Relative Ranking Vector calculated from the above step, the relative ranking of all the services for one sub-level attribute called the Eigen value is estimated using the instruction given in 11 and 12 .Finally the relative ranking vector is aggregated for each sub attribute at each level and the final relative ranking vector is computed.The Final Relative Ranking vector is sorted and the sorted list of cloud renderfarm services according to their ranks is obtained.However, the five QoS attributes considered in this work do not have any sub levels.

SAW Method of Ranking
The Simple Additive Weighting (SAW) method also known as the WSM (Weighted Sum Model) or the weighted linear Combination method or SM (Scoring method) calculates the overall score of a cloud render farm service by calculating the weighted sum average of all the attribute values.
SAW calculates an evaluation score for every alternative by multiplying the relative importance weights directly with the normalized value of the criteria for each alternative assigned by the user.The obtained product value is then summed up.The alternative service with the highest score is selected as the best cloud renderfarm service.
The formula to calculate the overall score (S i ) for an (N) alternative service with (M) QoS attributes is given below in Equation 1. 1)For i = 1, 2, 3,… N Where r ij refers to the normalized rating, 'i' indicates the i th alternative and' j' indicates the j th criterion.w j is the j th criterion weight.The formula to calculate benefit criteria value of r ij is given below by the equation 2. 2) Similarly, the formula to calculate the worst criteria value of r ij is given by equation 3.
(Equation 3) Where, x ij represents the original value of the j th criterion of the i th alternative.

Performance Comparison of Ranking Algorithms
To compare the performances of the two algorithms four different simulations with four different criteria of importance for QoS attributes were set as given above in the Table 1.The QoS details of the five attributes selected are collected for 3 real time cloud renderfarm services.The detail regarding the service providers are not provided to avoid the hindrance that may be caused to the service providers.In the simulation 1, the two QoS attributes or criteria namely the Render Node Cost and the File Upload Time are given more preference than the other criteria as given in Table 2.In Simulation 2, the Service Response Time (SRT) attribute is given more emphasize than any other QoS attribute as given in Table 3.In Table 4, it is worthy to note that, the criteria emphasized in Simulation 1 and 3 are similar as the two criteria namely the render node cost and the File upload time are given more preference than the other criteria.In Simulation 4, the file upload time is given more importance than the other QoS attributes as given in Table 5.

Results and Discussion
The performance of three real time cloud renderfarm services are ranked and compared based on five Quality of Service (QoS) attributes that are important to these services namely the Render Node Cost, File Upload Time, Availability, Elasticity and Service Response Time.The performance of these cloud renderfarm services are ranked in four different simulations by varying the     weights assigned for each QoS attribute and the ranking obtained are compared.The AHP method based ranks for Renderfarms is given in Figure 2 and the ranking of the renderfarms using the SAW method is given in Figure 3.
It can be clearly seen that, in Simulation 1 and 3, the RF2 is selected as the best alternative to achieve the criteria of low render node cost and low File upload time by both the algorithms.In simulation 2, the lowest Service Response Time criterion is best achieved by RF1 as ranked best service by both AHP and SAW methods.Since in Simulation 4, the file upload time is taken as the prime criteria the RF3 renderfarm service is selected as the best alternative with lowest file upload time by both AHP and SAW.
From the above discussions, it is evident that both AHP and SAW has similar ranking for each simulation criteria discussed and the ranking values are also very close to each other.

Conclusion and Future Work
This work compared and analyzed the raking of cloud renderfarm services using two Multi Criteria Decision Making (MCDM) methods.It was observed that the ranking value differs based on the weights assigned to each QoS attribute.The best cloud renderfarm service differs based on the QoS criteria weights.However, both AHP and SAW has similar ranking for each simulation criteria discussed and the ranking values are also very close to each other.Thus SAW method is a good alternative for the AHP and could be preferred instead of AHP when no hierarchy of attributes exists as in the case discussed in this work.However, if there many hierarchy levels with sub attribute then AHP qualifies as a better method to find out the aggregated rank value.In future, the QoS attributes that are more relevant to the cloud based rendering would be identified and the ranking would be simulated with different criterion using various other MCDM algorithms and the results would be compared to identify the efficiency and significance of the MCDM methods.

Figure 1 .
Figure 1.AHP Hierarchy of QoS Attributes for Cloud Renderfarm Services.

Table 1 .
Weight Assigned to QoS Attributes for each Simulation

Table 2 .
Ranking based on Simulation 1 attribute weights

Table 3 .
Ranking based on Simulation 2 attribute weights

Table 4 .
Ranking based on Simulation 3 attribute weights

Table 5 .
Ranking based on Simulation 4 attribute weights