HCI Exercise 1

Human-Computer Interaction SS 2024

Exercise 1: Heuristic Evaluation Plan (HE Plan)

First of all, read the sections in the lecture notes on heuristic evaluation. You will have to explain what this technique involves to your client (tutor) at the first client meeting.

Note: This exercise only entails writing the plan, you do not do the actual evaluation yet!

1 Plan Your Heuristic Evaluation

Prepare as follows:

  1. The four (or three) members of your group will be the evaluators for the heuristic evaluation.

  2. The web site which has been assigned to your group will be visible in TUGRAZonline in the comment field next to your group number.

  3. Unless otherwise notified, assume that you will be evaluating the entire web site.

    Generally speaking, do not follow links to external web sites and evaluate them too. However, it often makes sense to follow links to subsites and consider them too (say, tickets.site.com). Check with me or your tutor if you are unsure.

  4. Some of the chosen web sites are available in both English and German. Sometimes, some of the web sites are available in German only or in English only.

    If the web site has both English and German versions, each evaluator should pick one language (English or German) for the bulk of their evaluation, but then also take a look with the other language too. Unless directed otherwise by your tutor.

  5. Consider the target user population(s) for your assigned web site. Are there distinct user groups with distinct needs and goals?

  6. If the web site has several distinct user groups, you should consider all of the potential user groups in the heuristic evaluation.

  7. Discuss the goals the users from these groups might have and typical tasks they might want to perform on the web site.

  8. Adapt the HE Plan materials as necessary for your evaluation (see below).

  9. Plan for all of your evaluators to work at around the same time, to minimise the risk that the web site changes in between evaluations.

2 Write the HE Plan

Write a short plan for the heuristic evaluation:

  1. Some title information indicating the group number, the title/topic of the evaluation, and the name of each group member.

  2. A description of the methodology behind a Heuristic Evaluation, in your own words.

  3. Adapt the citations and references as instructed in the comments.

    See my Guide to Citations and References.

  4. User profiles describing which groups of users you think are supposed to be addressed by the web site. Describe the goals and typical tasks for each user group.

  5. The names, characteristics, and environments of each evaluator.

  6. Coordinate between yourselves, so that the evaluators use a mix of devices and browsers to inspect the web site:

    • Two of the evaluators should evaluate on a mobile device (= smartphone or tablet, running Android or iOS), in portrait orientation, each using a different approved mobile web browser.

      On mobile, the approved browsers are: Chrome, Firefox, Safari, and Samsung Internet Browser.

    • The other evaluator(s) should evaluate on a PC (= desktop or laptop, running Windows, MacOS, or Unix), in landscape orientation, each using a different approved PC web browser.

      On PC, the approved browsers are: Chrome, Firefox, and Safari.

    It is OK for one evaluator to use (say) Chrome on PC, and another evaluator to use Chrome on mobile, since they are different devices.

    You must use real devices and not an emulator or simulator.

    If you are using a mobile browser and your web site suggests you should go to the mobile version of the site, then do so. However, do not install a native app, even if your web site suggests you should do so.

  7. In an attempt to reflect real usage patterns, one of the evaluators on mobile and one of the evaluators on PC should plan to use an ad blocker.

    If you find a problem which may be due to blocked content, unblock ads temporarily to see whether the problem persists. If the problem then goes away, document it as a problem apparently caused by ad blocking.

  8. Before you start evaluating, plan to reset the browser to a fresh state:

    • Make sure all add-ons and extensions are disabled (except for an ad blocker, if you are using one via an extension).

    • Delete all cookies.

    • Delete all temporary files (clear the cache).

  9. Plan to try out your assigned web site first with all cookies enabled and then with most cookies disabled (only necessary cookies enabled) to see if there is any difference.

    If you find a problem which appears to be due to disabling cookies, document it as such.

  10. Decide how you will make make screen video recordings on the various platforms you have chosen to evaluate. See my Guide To Session Capture on various platforms. You should plan to record your voice (with a live commentary while you work) as well as what happens on screen.

    On touch devices, see if your recording software can turn on display of touch events.

    The screen recording should be at most at FullHD resolution (1920×1080 pixels in landscape orientation or 1080×1920 pixels in portrait orientation). If your device has higher resolution than FullHD, please plan to adjust the settings or make the browser window smaller. If your device has lower resolution than FullHD, record at its maximum resolution. The standard browser GUI should be included in the recording.

    If the native resolution of your device is too high in one or both directions, and you cannot adjust the settings or make the browser window smaller, you will have to record at higher pixel resolution and later transcode down uniformly to at most FullHD. Do not distort the video and do not create black or empty strips to either side (or top and bottom). See my Guide to Video Transcoding. Unfortunately, transcoding always results in a loss of quality.

    Use a good microphone, and make a test recording to make sure everything is working and the audio can be heard.

    Our preferred video format for screen recordings is MP4 with H.264 video and AAC audio. If your device/software can record in that format natively, that will be perfect. Otherwise, you will have to look at converting/transcoding your video clips after extraction.

    Plan to use the following recommended video settings:

    Output Video Resolution:1920×1080 (FullHD) [or 1080×1920] or lower
    Frame Rate:20
    Rate Control:VBR (Variable Bit Rate)
    Bit Rate:5000 Kbps (= 5 Mbps)

    Plan to use the following recommended audio settings:

    Codec:AAC (= mp4a)
    Sample Rate:44100 Hz
    Bit Rate:160 Kbps
    Bits per Sample:32
  11. Decide how you will extract video clips from the screen capture videos. See my Guide to Video Editing.

    Every finding must be illustrated with a video clip.

  12. Adapt and include the log files you will be using for the evaluation:

    • Assign each evaluator a two-letter shorthand code based on their initials. For example, Keith Andrews would be “ka” in lower case and “KA” in upper case.

      If two evaluators in your team both have the same initials, then assign one of them a variant. For example, Ken Anderson might be assigned "kn" in lower case and “KN” in upper case.

    • Copy the skeleton log file log-ee.txt from the materials to create a plain text log file for each evaluator:


      where ee is replaced by the lower case initials of the evaluator. Do not use any upper case letters, special characters, umlauts, or spaces in file names.

      For example, if your name is Keith Andrews, the log files should be named log-ka.txt.

    • Each evaluator should fill in the metadata at the start of their log file to match their (planned) evaluation environments. Enter the name of the corresponding evaluator. Take care to preserve the character encoding of the log files as UTF-8.

    • Each negative finding (problem) will be assigned an ID of the form EE-Neg01, EE-Neg02, etc. Each positive finding will be assigned an ID of the form EE-Pos01, EE-Pos02, etc. In each log file, replace the initials EE in the example problem and positive finding IDs with the upper case initials of the corresponding evaluator.

For the HE Plan:

3 Prepare the Presentation

  1. Prepare a 5-minute slide presentation of your HE Plan to show to your client at the client meeting (M1):

    • using slides (screenful by screenful).
    • using keywords and bulleted lists (not full sentences).

    Make a separate presentation. Do not simply open your HE Plan in a web browser and project that.

  2. You may prepare your slide presentation in Powerpoint, Keynote, HTML, or PDF as long as you can export it (without loss of fidelity) to PDF.

    Export your slide presentation to PDF and name it heplan-slides.pdf

    Open your PDF slide presentation with a PDF Viewer and make sure it can be used to give a presentation.

4 Prepare the HE Plan Directory

  1. Make a directory gT-GG-heplan for your heuristic evaluation plan, where T-GG is the number of your tutor and group. For example, g1-05-heplan for Group 1-05.

  2. Include your main file heplan.html.

  3. Copy over the files heuristics.pdf and report.css unchanged.

  4. Create a subdirectory logs for your skeleton log files. For each evaluator, there should be one log file named log-ee.txt, where ee are the lower case initials of that evaluator.

  5. Create a subdirectory presentation for your presentation slides, heplan-slides.pdf.

  6. When naming your files and directories, use only lower case letters, digits, and hyphens, from the 7-bit ASCII character set.

    Do not use upper case letters, spaces, underscores, umlauts, or special characters. Only use dot (.) to designate a file's extension, nowhere else in a file name and nowhere in a folder name.

  7. Your directory structure should look something like this:

  8. Tidy up your directory. Do not leave junk files, backup files, etc. lying around.

  9. Your directory must occupy 20 MB (20,000,000 bytes) or less. Check the size before uploading.

    If your files are too big, you may have to delete something or reduce the size of something. Sometimes, you can significantly reduce the size of your slides PDF, without significantly affecting quality. Contact your tutor if you are unsure what to remove or make smaller.

  10. Make a zip file of your directory and keep it somewhere else as a backup.

    Do not upload the zip file to Sapphire.

5 Upload Your HE Plan to Sapphire

  1. Submit your HE Plan before the deadline to the Sapphire online submission system:


    See my Guide to Using Sapphire.

  2. Ex 1 is a group exercise. The group makes a single submission, as a group.

    Every member of the group has access permissions to create and modify the group's submission, but it is best to designate a single person to do the uploading or any modifications, in order to avoid potential conflicts with other group members making changes in parallel.

  3. Use the following approach to upload your files:

    1. Go to Exercises and select “Ex 1: HE Plan”.

    2. Create a submission for Ex 1.

    3. Click on Upload.

    4. Drag your local submission folder gT-GG-heplan/ (yes, the folder, for example g1-05-heplan/, not all of the individual files) and drop it into the upload area.

      Do not upload your submission as a zip file.

    5. Watch while the files upload (any sub-folder structure will be created appropriately).

    6. Check your submission folder to make sure everything is there and looks right.

  4. If you make any changes (including deleting files) to your submission after the deadline, your submission will be flagged with the new timestamp and will be considered to be a late submission with the corresponding points deduction.

  5. Submissions will cease to be accepted 48 hours after the deadline.

  6. At the client meeting, you must present the same version of your work which was handed in (uploaded to Sapphire).