HCI Exercise 2b

(INB.05021UF Human-Computer Interaction 3VU SS 2018)

Exercise 2b: Heuristic Evaluation (HE Report)

1 Aggregate the Individual Findings

  1. Assemble the individual log files and associated video clips and screenshots from each evaluator in your group.

  2. As a group, form a combined (aggregated) list of problems (and a combined list of positives) using the spreadsheet helist.xls (or helist-de.xls for German content) as a template:

    • If two or more evaluators find the same (or very similar) problems, combine them into one.

    • List many small related issues (such as 27 individual typos) as one problem (rather than 27 problems).

    • Indicate which problems were found by which evaluator.

    • For each problem, determine whether the problem is general to all browsers and platforms or specific to a particular browser or platform. If a problem is specific, indicate the browser or platform where it occurs. If a problem is general, leave the corresponding table cell empty.

    Proceed analogously for the positive findings.

  3. Individually (not working together), assign severity ratings to each problem. Use the 0--4 integer severity scale below:

    SeverityMeaning
    4Catastophic Problem
    3Serious Problem
    2Minor Problem
    1Cosmetic Problem
    0Not a Problem

    Individual ratings must be integers, fractional ratings are not allowed.

  4. Individually (not working together), assign positivity ratings to each positive finding. Use the 0--4 integer positivity scale below:

    PositivityMeaning
    4Extremely Positive
    3Major Positive
    2Minor Positive
    1Cosmetic Positive
    0Not a Positive

    Individual ratings must be integers, fractional ratings are not allowed.

  5. Calculate the mean severity rating for each problem to 2 decimal places. Calculate the mean positivity rating for each positive finding to 2 decimal places.

  6. Sort the problems into descending order of average severity (most severe first). Sort the positives into descending order of average positivity (most positive first).

  7. Renumber the problems and positives so that the finding at the top of each list is number 1.

  8. For each finding, select the best (most representative) available video clip or screenshot to illustrate the finding.

    In certain cases, it might make sense to select two or three video clips or screenshots to illustrate one finding, but you do not have to list and include every single video clip and screenshot from every evaluator for every finding.

  9. Rename the selected video clips and screenshots to insert the new number immediately after the first letter (n or p) in the file name. For example, p01-ka-breadcrumbs.mp4.

    If you decide to include two or three video clips or screenshots from the same evaluator for the same finding, then add a digit after the keywords, for example: n01-ka-screen-1.jpg and n01-ka-screen-2.jpg.

  10. Transfer the sorted lists of problems and of positives into simple Polyglot XHTML5 tables.

    Do not just save the spreadsheet from Excel as HTML, since this creates bloated HTML code, which is probably not Polyglot XHTML5.

    For example, save the spreadsheet as a CSV (comma seperated value) file and then use a text editor to manipulate it:

    • Replace the commas with cell boundaries (</td><td>).
    • Prepend an initial <tr><td> to the first cell and append a final </td></tr> to the final cell on each row.

    If you know perl, you might also use a Perl script to generate simple XHTML5 table entries from a CSV file. I have an example Perl script (rename it to csv2html.pl).

2 Write the Heuristic Evaluation Report

Write a report of the heuristic evaluation using the skeleton report provided:

  1. Some title information indicating the group number, the title/topic of the evaluation, and the name of each group member.

  2. A summary of the evaluation and the most important findings written for your client's management. In real life, a manager would typically not have time to read the whole report, but would read only the executive summary to find out how the evaluation was done and what the main findings were.

    The description of the evaluation scope and procedure should take up no more than 25% of the executive summary. Most of the executive summary should summarise the main findings.

  3. A description of the methodology behind a Heuristic Evaluation, in your own words. Assumptions made about user characteristics and groupings and the extent of the evaluation.

  4. A description of the environment (hardware, browser version, OS, date of evaluation, etc.) used by each evaluator.

    Explain how each evaluator extracted video clips and screenshots from their screen capture video on each particular evaluation device or platform.

  5. Discussion of three most positive findings, each illustrated with a video clip or screenshot.

  6. Aggregate table of positive findings in descending order of positivity.

  7. Analysis and discussion of the five most severe problems (the top five from your sorted list), each illustrated with a video clip or screenshot.

  8. Aggregate table of problems in descending order of severity.

For the HE Report:

3 Prepare Your HE Report Directory

4 Upload Your HE Report to Sapphire