Revisiting Heuristic Evaluation Methods to Improve the Reliability of Findings
نویسندگان
چکیده
The heuristic evaluation (HE) method is one of the most common in the suite of tools for usability evaluations because it is a fast, inexpensive and resource-efficient process in relation to the many usability issues it generates. The method emphasizes completely independent initial expert evaluations. Inter-rater reliability and agreement coefficients are not calculated. The variability across evaluators, even dual domain experts, can be significant as is seen in the case study here. The implications of this wide variability mean that results are unique to each HE, results are not readily reproducible and HE research on usability is not yet creating a uniform body of knowledge. We offer recommendations to improve the science by incorporating selected techniques from qualitative research: calculating inter-rater reliability and agreement scores, creating a codebook to define concepts/categories and offering crucial information about raters' backgrounds, agreement techniques and the evaluation setting.
منابع مشابه
Revisiting the Relationship Between Systems of Innovation and Health Systems: A Response to Recent Commentaries
متن کامل
Design and Psychometrics of a Questionnaire for Assessing the Performance of Faculty Members in Providing Virtual Education during the Covid-19 Pandemic
Background and Purpose: Given the Covid-19 and immediate investment in providing virtual university education, it is important to measure the performance of faculty members in this area. Accordingly, this study was conducted to design and psychometrics of a questionnaire to assess the performance of faculty members in providing virtual training. Material and Methods: A descriptive study of ins...
متن کاملDeveloping and Psychometrics "Scale of Performance Evaluation Criteria for University Faculty Members"
Introduction: Evaluation of university teachers is a necessity in order to improve the knowledge of the community. In this regard, the performance evaluation instruments of university faculty members are different. Therefore, the aim of present study was to develop and Psychometrics "Scale of Performance Evaluation Criteria for University Faculty Members". Methods: The present study is of metho...
متن کاملHeuristic approach to solve hybrid flow shop scheduling problem with unrelated parallel machines
In hybrid flow shop scheduling problem (HFS) with unrelated parallel machines, a set of n jobs are processed on k machines. A mixed integer linear programming (MILP) model for the HFS scheduling problems with unrelated parallel machines has been proposed to minimize the maximum completion time (makespan). Since the problem is shown to be NP-complete, it is necessary to use heuristic methods to ...
متن کاملTowards Increasing Reliability of Heuristic Evaluation
To date comparative studies have not convinced the HCI community about the reliability of results from usability evaluations. User testing of the same products by different usability teams found different problems (Molich et al., 1999). Discount methods also have the same problem. With heuristic evaluation method, for example, overlaps between evaluators’ results are often as low as 10% (Molich...
متن کاملذخیره در منابع من
با ذخیره ی این منبع در منابع من، دسترسی به آن را برای استفاده های بعدی آسان تر کنید
برای دانلود متن کامل این مقاله و بیش از 32 میلیون مقاله دیگر ابتدا ثبت نام کنید
ثبت ناماگر عضو سایت هستید لطفا وارد حساب کاربری خود شوید
ورودعنوان ژورنال:
- Studies in health technology and informatics
دوره 205 شماره
صفحات -
تاریخ انتشار 2014