Abstract
In a competition, a number of evaluators are responsible for reviewing and ranking a group of participants. After reviewing all participants by each evaluator, the final ranking result is obtained by averaging all evaluator’s ranking, which is called “exhaustive ranking”. However, if the number of participants is large, an evaluator is not able to review all participants. Thus, the participants have to be divided into several subgroups, and an evaluator only needs to review a subgroup. The final ranking result is therefore determined by the partial evaluations by various evaluators. The focus of this study is, without increasing the efforts of evaluators, how to determine a ranking as close to exhaustive ranking as possible under the condition that an evaluator can only review a portion of all participants. The proposed evaluation procedure is called cooperative ranking methods that involve a number of iterations. In an iteration, all participants are divided into a number of subgroups and each subgroup is reviewed by an evaluator. After being evaluated in previous iterations, a participant could be re-assigned to a different subgroup based on the result of previous iterations. By utilizing the previous evaluation results, the participants would be evenly assigned to different subgroups. The experiment results show that in a competition of 300 participants who are divided into three subgroups, the average error in ranking position of a participant is 4.18 in three iterations by applying the proposed evaluation method. On the other hand, a traditional approach that does not adopts regrouping, the average error is as high as 11.88. In addition, in actual application of the proposed evaluation procedure whose exhaustive ranking is unknown, the method to estimate the error in ranking position is also proposed in this study. Keywords: ranking, large scale competition, regroup, cooperative, group.