Each evaluator must inspect the interface alone.
Set up screen and audio capture on your device. See my Guide to Session Capture on various platforms. Use a good microphone, and make a test recording to make sure everything is working and the audio can be heard.
Do not use recording software which leaves behind a watermark. On mouse devices, turn on recording of the mouse pointer (even make it slightly larger), but turn off any mouse trails. On touch devices, see if your recording software can turn on display of touch events, but turn off touch trails.
The screen recording should be at most at FullHD resolution (i.e. 1920×1080 pixels in landscape orientation, or 1080×1920 pixels in portrait orientation). If your device has higher resolution than FullHD, please plan to adjust the settings or make the browser window smaller. If your device has lower resolution than FullHD, record at its maximum resolution. The standard browser window and GUI should be included in the recording. Other things outside the browser window should not be included in the recording.
If the native resolution of your device is too high in one or both directions, and you cannot adjust the settings or make the browser window smaller, you will have to record at higher pixel resolution and later transcode down uniformly to at most FullHD. Do not distort the video and do not create black or empty strips to either side (or top and bottom). See my Guide to Video Transcoding. Unfortunately, transcoding always results in a loss of quality.
Record in MP4 format with H.264 video and AAC audio if possible. If your device/software can record in that format natively, that is perfect.
Otherwise, you will have to convert/transcode your video clips to MP4 (with H.264 video and AAC audio) format later on. Unfortunately, transcoding always results in a loss of quality.
Use the following recommended video settings:
| Container: | MP4 |
| Output Video Resolution: | 1920×1080 (FullHD) [or 1080×1920] or lower |
| Frame Rate: | 20 |
| Codec: | H.264 |
| Rate Control: | VBR (Variable Bit Rate) |
| Bit Rate: | 5000 Kbps (= 5 Mbps) |
Use the following recommended audio settings:
| Channels: | Stereo |
| Codec: | AAC (= mp4a) |
| Sample Rate: | 44100 Hz |
| Bit Rate: | 160 Kbps |
| Bits per Sample: | 32 |
For this heuristic evaluation report, do not record the face of the evaluator in any video recording, only the screen and audio.
Keep in mind the target user population(s) and their typical tasks.
Print out an A4 copy of the Andrews General Usability Heuristics provided in the materials.
Most of the chosen web sites are available in German. Some of the chosen web sites are available in English and German. Sometimes, some of the web sites are available in English only.
If the web site has both English and German versions, each evaluator should pick one language (English or German) for the bulk of their evaluation, but then also take a look with the other language too.
Speak English for the audio commentary while evaluating, regardless of the language of the web site.
Use the device and browser combinations you specified in your HE Plan, so that the evaluators use a mix of devices and browsers to inspect the web site:
Two of the evaluators should evaluate on a mobile device (= smartphone or tablet, running Android or iOS), in portrait orientation, each using a different approved mobile web browser.
On mobile, the approved browsers are: Chrome, Firefox, Safari, and Samsung Internet Browser.
The other evaluator(s) should evaluate on a PC (= desktop or laptop, running Windows, MacOS, or Unix), in landscape orientation, each using a different approved PC web browser.
On PC, the approved browsers are: Chrome, Firefox, and Safari.
It is OK for one evaluator to use (say) Chrome on PC, and another evaluator to use Chrome on mobile, since they are different devices.
You must use real devices and not an emulator or simulator.
If you are using a mobile browser and your web site suggests you should go to the mobile version of the site, then do so. However, do not install a native app, even if your web site suggests you should do so.
As specified in your HE Plan, one of the evaluators on mobile and one of the evaluators on PC should use an ad blocker.
If you find a problem which may be due to blocked content, unblock ads temporarily to see whether the problem persists. If the problem then goes away, document it as a problem apparently caused by ad blocking.
Remember to reset the browser to a fresh state before you start evaluating:
Make sure all add-ons and extensions are disabled (except for an ad blocker, if you are using one via an extension).
Delete all cookies.
Delete all temporary files (clear the cache).
Try out your assigned website first with all cookies enabled and then with most cookies disabled (only necessary cookies enabled) to see if there is any difference.
If you find a problem which appears to be due to disabling cookies, document it as such.
Start the screen video (+ audio) capture, making sure to hide any UI controls of the recording software.
To be clear, the evaluator should record their entire inspection session, so that it is available to cut out video clips from, in case some findings cannot later be reconstructed.
Inspect the interface, talking out loud and also noting problems and positive impressions in chronological order of discovery in the plain text log file you prepared previously.
The log file should be called log-ee.txt
or log-ee.txt, where ee are your initials in
lower case. Use the templates you prepared in Ex1 HE Plan.
Each negative finding (problem) is assigned an ID in
chronological order of discovery of the form
EE-Neg01, EE-Neg02, etc.,
where EE are the evaluator's initials in upper case.
Analogously for positive findings: EE-Pos01,
EE-Pos02, etc.
For each finding, enter the following information into the log file:
EE-Neg01.
ee-neg01-keywords.mp4.
In addition, for negative findings (problems), also make a note of which heuristic, if any, each problem falls under:
Most problems will fall under one of the heuristics, but it is OK if some of the problems you report are not covered by a heuristic (leave the heuristic field in the log file blank). We will not assign heuristics to positive findings in this evaluation, only to negative findings.
Once you have finished evaluating, save your screen video (+ audio) recording and your log file.
Make sure your log file is a plain text file encoded in UTF-8, of at most 100 kb.
Make sure that when editing the log file, that when you save it, it is still in UTF-8 format. Check the settings in whatever plain text editor you are using.
If there are many examples of the same general problem (for example, typos across several pages), count this as one problem and give two or three examples in the description.
In total, we would generally expect each individual evaluator to find between 10 and 20 problems and at least 3 positives on each device, depending on the quality of the web site you are evaluating.
Re-create a short video clip to illustrate each finding from each device.
Every finding must be accompanied by at least one video clip. (sometimes two, say with and without cookies, if that is what best illustrates the problem).
Each video clip must contain an audio track in English, concisely describing the finding.
Where a finding cannot be recreated, extract a video clip from your evaluation recording.
If the existing audio commentary does not describe the finding adequately, replace the audio commentary with a new one.
Make sure the audio levels are high enough, so that your audio commentary can be clearly heard.
Each video clip should be long enough to illustrate the finding, but no longer. Typically, video clips are 10 to 15 seconds. At a maximum, each video clip should be no longer than 20 seconds duration.
In some rare cases, say to illustrate extremely slow loading time, it might be tempting to include a video clip longer than 20 seconds. In such a case (for the purposes of this course), edit the video clip to show say an inital 8 seconds, then a transition frame something like “28 seconds later” for 2 seconds, then the final 8 seconds (18 seconds duration in total).
Each video clip should be no larger than FullHD resolution (1920×1080 pixels in landscape orientation or 1080×1920 pixels in portrait orientation).
Otherwise, you will have to transcode down uniformly to at most FullHD. Do not distort the video and do not create black or empty strips to either side (or top and bottom). See my Guide to Video Transcoding. Unfortunately, transcoding always results in a loss of quality.
Each video clip should be no larger than 10 MB in size!
Otherwise, make it shorter in duration if possible, or else transcode to a lower video bitrate (with the resulting loss of quality).
Name your video clips according to the following naming scheme:
EE-negXX-keywords.mp4
EE-posXX-keywords.mp4
where EE is replaced by the lower case initials of the
evaluator (e.g. ka for Keith Andrews), neg
indicates a negative finding (problem) and pos indicates
a positive finding,
XX is the two-digit number in chronological order of
discovery, and keywords comprises one or more words
describing the finding. If you need to, you can use upto four keywords
separated by hyphens.
For example:
ka-neg01-links-yellow.mp4
ka-pos01-breadcrumbs-work-well.mp4
...
For file and folder names, use only lower case letters, digits, and
hyphens, from the 7-bit ASCII character set. Do not use any upper case
letters, spaces, underscores, or special characters in either the name
or extension. Under windows, turn on the display of file extensions,
so that you see them! The name ka_neg01_links_yellow.MP4
does not conform to the naming scheme.
Make a directory called gT-GG-EE-eval for your individual
evaluation files, where T-GG is the number of your tutor and
group and EE are your evaluator initials in lower
case. For example, g1-05-ka-eval for Group 1-05 and
evaluator Keith Andrews.
Place your individual evaluation log file and your video clips inside this directory.
When naming your files and directories, use only lower case letters, digits, and hyphens, from the 7-bit ASCII character set.
Do not use upper case letters, spaces, underscores, umlauts, or special characters. Only ever use dot (.) to designate a file's extension, nowhere else in a file name, and nowhere in a folder name.
Your directory structure should look something like this:
g1-05-ka-eval/ log-ka.txt ka-neg01-links-yellow.mp4 ka-neg02-fuzzy-background.mp4 ... ka-pos01-breadcrumbs-work-well.mp4 ka-pos02-good-site-map.mp4 ...
Tidy up your directory. Do not leave junk files, backup files, etc. lying around.
Your directory must occupy 200 MB (200,000,000 bytes) or less. Check the size before uploading.
If your files are too big, you may have to delete something or reduce the size of something. Contact your tutor if you are unsure what to remove or make smaller.
Make a zip file of your directory and keep it elsewhere as a backup (do not upload it).
Submit your evaluation files before the deadline to the Sapphire online submission system:
https://sapphire.isds.tugraz.at/
Ex 2 is an individual exercise. Every member of the group makes their own individual submission.
Use the following approach to upload your files:
gT-GG-EE-eval/ (yes, the
folder, for example g1-05-ka-eval/, not all of the individual
files) and drop it into the upload area.
See my Guide to Using Sapphire.
Submissions will cease to be accepted 48 hours after the deadline.