Revisiting Heuristic Evaluation Methods to Improve the Reliability of Findings

نویسندگان

  • Mattias Georgsson
  • Charlene R. Weir
  • Nancy Staggers
چکیده

The heuristic evaluation (HE) method is one of the most common in the suite of tools for usability evaluations because it is a fast, inexpensive and resource-efficient process in relation to the many usability issues it generates. The method emphasizes completely independent initial expert evaluations. Inter-rater reliability and agreement coefficients are not calculated. The variability across evaluators, even dual domain experts, can be significant as is seen in the case study here. The implications of this wide variability mean that results are unique to each HE, results are not readily reproducible and HE research on usability is not yet creating a uniform body of knowledge. We offer recommendations to improve the science by incorporating selected techniques from qualitative research: calculating inter-rater reliability and agreement scores, creating a codebook to define concepts/categories and offering crucial information about raters' backgrounds, agreement techniques and the evaluation setting.

برای دانلود متن کامل این مقاله و بیش از 32 میلیون مقاله دیگر ابتدا ثبت نام کنید

ثبت نام

اگر عضو سایت هستید لطفا وارد حساب کاربری خود شوید

منابع مشابه

Design and Psychometrics of a Questionnaire for Assessing the Performance of Faculty Members in Providing Virtual Education during the Covid-19 Pandemic

Background and Purpose: Given the Covid-19 and immediate investment in providing virtual university education, it is important to measure the performance of faculty members in this area. Accordingly, this study was conducted to design and psychometrics of a questionnaire to assess the performance of faculty members in providing virtual training. Material and Methods: A descriptive study of ins...

متن کامل

Developing and Psychometrics "Scale of Performance Evaluation Criteria for University Faculty Members"

Introduction: Evaluation of university teachers is a necessity in order to improve the knowledge of the community. In this regard, the performance evaluation instruments of university faculty members are different. Therefore, the aim of present study was to develop and Psychometrics "Scale of Performance Evaluation Criteria for University Faculty Members". Methods: The present study is of metho...

متن کامل

Heuristic approach to solve hybrid flow shop scheduling problem with unrelated parallel machines

In hybrid flow shop scheduling problem (HFS) with unrelated parallel machines, a set of n jobs are processed on k machines. A mixed integer linear programming (MILP) model for the HFS scheduling problems with unrelated parallel machines has been proposed to minimize the maximum completion time (makespan). Since the problem is shown to be NP-complete, it is necessary to use heuristic methods to ...

متن کامل

Towards Increasing Reliability of Heuristic Evaluation

To date comparative studies have not convinced the HCI community about the reliability of results from usability evaluations. User testing of the same products by different usability teams found different problems (Molich et al., 1999). Discount methods also have the same problem. With heuristic evaluation method, for example, overlaps between evaluators’ results are often as low as 10% (Molich...

متن کامل

ذخیره در منابع من


  با ذخیره ی این منبع در منابع من، دسترسی به آن را برای استفاده های بعدی آسان تر کنید

برای دانلود متن کامل این مقاله و بیش از 32 میلیون مقاله دیگر ابتدا ثبت نام کنید

ثبت نام

اگر عضو سایت هستید لطفا وارد حساب کاربری خود شوید

عنوان ژورنال:
  • Studies in health technology and informatics

دوره 205  شماره 

صفحات  -

تاریخ انتشار 2014