First of all, read the sections in the lecture notes on usability testing and thinking aloud tests.
For this exercise, your group will continue to play the role of user interface consultants who have been contracted to evaluate a web site. The idea is to identify as many potential usability problems as possible and feed this input into the next design phase.
Your tutor will play the role of your client (the manager of the web site you are evaluating). You will present the results of your thinking aloud test to your client at Meeting M3 at M3 Slots.
Use the report template and materials provided in the Ex3a Materials.
For each web site, the user group to test, the languages available to test, and the internal task list are specified in the Ex3a Tasks.
Run all of your tests in the same language (either English or German):
If you are testing a site in English, choose test users who speak English, and run the test in English. User-facing materials should be in English, and the users should think out loud in English.
If you are testing a site in German, choose test users who speak German, and run the test in German. User-facing materials should be in German, and the users should think out loud in German.
If your web site is listed as available to test in both English and German, choose one of the two languages to test. All of your tests must be run in the same language!
Organise five test users from the specified user group, one user for the pilot test and four users for the real test.
For these practicals we have tried to select as broad a user group as possible, to make it easier for you to find potential test users. Try to find test users who broadly match the chosen user group, otherwise get as close as you can, given that we have no budget available and are dependent on the good will of your friends, family, and colleagues.
The pilot user should come from the middle of the user group (not the extremes).
The four users for the real test should span the range of the user group (as far as is possible).
Assign (sensible) first name aliases to your test users, different from their real first names.
When selecting test users in the real world, you typically ask them some questions from a screening questionnaire to make sure they fit the profile of the user group you are testing.
For these practicals, make sure that your test users meet the following criteria:
At least 18 years of age.
Native or proficient speaker of German (if you are running your tests in German) or English (if you are running your tests in English).
Experienced in using both computers and the web.
Not currently taking the HCI course this semester.
Not serving as a test user for another HCI group this semester.
Organise a quiet room with a PC (= desktop or laptop, running Windows, MacOS, or Unix) and an internet connection.
It is much more convenient in terms of scheduling for you, your test users, and us, if you can arrange your own test room. If you cannot, a limited number of slots are available for a room at the institute. See the Kit and Room Slots.
Plan to leave one full day between pilot test and real tests.
Plan to run the tests with one of Chrome, Firefox, or Safari on a PC. All test users must use the same device and browser!
Set the browser to light mode (dark text on a light background).
Chrome: More Tools → Customise Chrome → Appearance: Light
Firefox: Tools → Settings → General → Language and Appearance: Light
For these tests, do not use an ad blocker.
Leave it up to each test user to decide what they do with cookies.
Use the same test environment (room, internet connection, device, browser, video equipment) for all five test users.
Unless, of course, you discover a problem and have to swap a piece of equipment.
It is OK for members of the test team to swap roles between tests (e.g. the facilitator for one test can be the video operator for the next test).
Each user will be asked to perform five tasks. See the internal task list for your web site. The tasks have the following structure:
Tasks 1, 2, 4, and 5 will be predefined by us. You will need to formulate your own Task 3.
Prepare external task slips to match the tasks for your test web site.
See the examples (external-tasks-ta.pdf) provided in the
materials.
A LaTeX template is provided in the materials for you to use if you wish.
For registration tasks, users may have to enter personal details, such as a name and address. They can either use their own real identity, if they are comfortable with that, or they can use an obviously fictitious identity (such as Max Mustermann, Musterplatz 1, 8010 Graz), as they please. You should prepare a fictitious identity for each test user in advance, to hand to the user when the time comes. This may also involve preparing free email addresses in advance, one for each test user to use.
For an ordering or submission process, the user should go through the process, up to the point where they are about to complete the order or submission (when you should stop them!). They should not actually go through with any order or submission, unless of course they are using their real identity and they really do want to do it.
Adapt the orientation script (orient.html) provided in
the materials for your test (first name of facilitator, first names of
test team, etc.).
If you are testing in German, use and adapt the German version of the
orientation script (orient-de.html).
Extend the background questionnaire (background.html)
provided in the materials with three domain-specific questions
applicable to your test web site.
If you are testing in German, use and extend the German version of the
background questionnaire (background-de.html).
Extend the standard questions in the interview script
(interview.html) provided in the materials to at least
four questions (two are already suggested).
If you are testing in German, use and extend the German version of the
interview script (interview-de.html).
Extend the feedback questionnaire (feedback.html)
provided in the materials with two domain-specific questions
applicable to your test web site.
If you are testing in German, use and extend the German version of the
feedback questionnaire (feedback-de.html).
You should plan to delete any personal data of the users (scans of consent forms, full video recordings, mapping of real names to ids and aliases) one year afterwards.
Backups must also be deleted: back up your video recordings in one or two well-defined locations, say on external hard disks, so you are in control of them and know where to find and delete them. Be careful with cloud backups, turn off synchronisation for the personal data. If you are recording the external video on a device with cloud sync, deactivate it for these recordings.
You must capture each test user's session on the PC using session capture software and a webcam. See my Guide to Session Capture. For these thinking aloud tests, you should capture usercam video of the user's face, in addition to audio and the screen.
For the session recordings:
Our preferred video format is MP4 (H.264 video with AAC audio) at FullHD (1920×1080) resolution. Record in that format if possible.
Otherwise, you will have to convert/transcode your video clips to MP4 (with H.264 video and AAC audio) format later on. Unfortunately, transcoding always results in a loss of quality.
The session recording should be at most at FullHD resolution (1920×1080 pixels) in landscape orientation. If your device has higher resolution than FullHD, please plan to adjust the capture settings or make the browser window smaller. If your device has lower resolution than FullHD, record at its maximum resolution.
If the native resolution of your device is too high in one or both directions, and you cannot adjust the settings or make the browser window smaller, you will have to record at higher pixel resolution and later transcode down uniformly to at most FullHD. Do not distort the video and do not create black margins or empty strips to either side (or top and bottom). See my Guide to Video Transcoding.
Do not use recording software which leaves behind a watermark.
The standard browser window and GUI should be included in the recording. Other things outside the browser window (taskbar, desktop icons, ...) should not be included in the recording, except possibly the webcam with the user's face.
Turn on recording of the mouse pointer (possibly even make it slightly larger), but turn off any mouse trails.
Use the following recommended video settings:
| Container: | MP4 |
| Output Video Resolution: | 1920×1080 (FullHD) |
| Frame Rate: | 20 |
| Codec: | H.264 |
| Rate Control: | VBR (Variable Bit Rate) |
| Bit Rate: | 5000 Kbps (= 5 Mbps) |
To capture user audio in the session video, use a built-in or external webcam microphone or an external microphone plugged into the device.
Use the following recommended audio settings:
| Channels: | Stereo |
| Codec: | AAC (= mp4a) |
| Sample Rate: | 44100 Hz |
| Bit Rate: | 160 Kbps |
| Bits per Sample: | 32 |
To capture the user's face in the session video, use a built-in or external webcam and place the picture, say, in the bottom right-hand or bottom left-hand corner of the screen, depending on what works best for your web site.
If your screen is large enough, you could also make the browser window a little narrower, and place the webcam in the margin to the right of the browser, at the bottom. This way, it will not overlap any content of the browser window.
In addition to session capture on the test device, you must record each test with an external tripod-mounted video camera (tripod, video camera, microphone, and mirror).
It is much more convenient in terms of scheduling for you, your test users, and us, if you can arrange your own external video recording equipment. If you cannot, a limited number of sets of video equipment will be available for loan, see the Kit and Room Slots.
For the external video recordings:
For the thinking aloud tests in this course, the external video camera must be used to record the entire test procedure for each user, from the moment the user enters the room to the moment the user leaves the room. This includes the user being greeted, being asked the questions on the background questionnaire, receiving training, performing the tasks, being interviewed after the final task, and filling out the feedback questionnaire. This applies both to the pilot test and the real tests!
If you can, greet the user at a separate table, setting up the external camera to initially point at the table (filming the introduction, background questionnaire, and consent form), then as the user moves over to the test PC, swing the camera around to shoot over the shoulder of the user for the actual test session.
Use a mirror next to the monitor or screen to capture the user's facial expressions.
Set up the field of view (recording area) to film over the shoulder of the user, so that the screen, keyboard, and the user's head in the mirror are framed in the picture.
You must use a tripod. Do not hand-hold the camera or move it around unnecessarily. Avoid excessive panning and zooming of the external video camera.
Use an external microphone to capture audio on the external video camera:
Use headphones to monitor the audio as it is being recorded. Make sure the audio of the test user is loud and clear.
You can try using the camera's built-in microphone, but since it will be located behind the test user, the audio of the test user might be too quiet (and the microphone may pick up other annoying background sounds too).
Our preferred video format for the external recordings is MP4 (H.264 video with AAC audio) at FullHD (1920×1080) resolution. Record in that format if possible.
If your external video camera cannot record in the above format, we
will try to play any of the following formats: MP4 with other video or
audio codecs (.mp4), MPEG-1 or MPEG-2
(.mpg), QuickTime (.mov), Matroska
(.mkv), and WebM (.webm). Please do not use
any obscure or proprietary video or audio codecs.
If we have difficulties playing your external video files, you may have to convert your videos into a different format and hand them in again.
Record in landscape mode (the video is wider than it is higher), not portrait mode.
Do not record at less than 1280×720 or higher than 1920×1080 (FullHD) resolution.
Turn off any data fields (date, time, mode, etc.) in the camera which would otherwise overlay the video recording.
Make a test recording to check how the camera works, how to select a format and resolution, that the external microphone works, the audio levels are high enough, and the type and size of video files produced.
First run a pilot test with one test user, taken from the middle of the target user group.
Leave at least one full day between pilot test and real tests, to make any changes which might be necessary.
Then, run the real test with 4 test users, spanning the target user group.
For each test user:
Greet the user and explain the purpose of the test using the orientation script provided (adapted to your web site).
Make sure the TPID of the test user (of the
form HCI-SSyyyy-Gx-xx-TPn, for
example HCI-SS2025-G1-01-TP1) is written on the top right
of the consent form.
Explain how you will deal with data protection: that you will delete any personal data of the users (scans of consent forms, full video recordings, mapping of real names to ids and aliases) one year afterwards.
Ask the user to read and sign the consent form provided. Answer any questions they may have.
Provide the user with a copy of the form, or let them take a photo of the form, if they wish.
Ask the user the questions on the background questionnaire. The facilitator should hold the pen and write the answers in the form, not the user.
Go to uncut.at
and find out which films are on tonight at the Royal English
Cinema, whilst thinking aloud yourself and explaining everything you do.
Or, show the user a short video clip of a previous user who is giving lots of good feedback (on a different interface). You may use this video clip (13,244,026 bytes), if you wish.
Reset the browser to a fresh state for each new user:
Make sure all add-ons and extensions are disabled.
Delete all cookies.
Delete all temporary files (clear the cache).
Delete any previous browsing history.
Set the page being displayed to google.com.
Run the test using the tasks provided:
Provide the tasks on individual task slips, one by one.
The facilitator should read out the task slip and then place on the table next to the computer.
The scribe should take notes on paper or in a text file or spreadsheet on a laptop.
Interview the user, beginning with the open question: "So, how was it?". Then ask your pre-prepared interview questions.
After the interview, ask the user to fill out the feedback questionnaire provided. The user should fill out the feedback questionnaire themselves (the facilitator hands them the pen).
Note:
Use the spreadsheet data-ta.xlsx provided in the
materials to collect and collate the results.
Enter the information from the background and feedback questionnaires into the spreadsheet.
Enter the task success metrics into the spreadsheet.
Analyse the session capture videos and collect a list of positive findings and negative findings (problems) in the spreadsheet. Note the timestamp where the finding started.
For each positive or negative finding, determine which test users experienced the finding. For each finding, create one video clip for every test user who experienced this finding to illustrate what happened. The video clips should be in MP4 (H.264 video and AAC audio) format and be no larger than FullHD (1920×1080) resolution; transcode if necessary. Each video clip should be no longer than 20 seconds in duration and no larger than 10 MB in size.
The short video clips illustrating findings are considered to be part of the report. Identifiable faces of the test users in the video clips must be made unidentifiable by blurring, pixelating, or blacking out. See my Guide to Face Blurring.
Since the test users are not identified by their real names in the report, it is probably not necessary to use a voice modifier to obscure their voices.
Initially name the video clips according to the following naming scheme:
n-tpY-keywords.mp4
p-tpY-keywords.mp4
where n indicates a negative finding (problem)
and p indicates a positive finding,
Y is replaced by the number of the test user
where the finding was observed, and keywords comprises
one to four keywords separated by hyphens to describe the finding. Use
only lower case letters, digits, and hyphens from the 7-bit ASCII
character set.
For example:
n-tp4-unclear-when-delivery-free.mp4
p-tp3-reg-clear.mp4
The finding number will be inserted later, once the severity (or
positivity) of all findings has been determined and the findings have
been sorted and ranked
(e.g. n01-tp4-unclear-when-delivery-free.mp4).
We would expect each group to find between 10 and 30 problems (over all test users), depending on the quality of the web site you are testing. We would expect each group to find at least 3 positives (over all test users).
Individually rate the problems by severity and the positive findings by positivity (working alone), then form a group average (as for the heuristic evaluation). Enter these into the spreadsheet and sort the tables in descending order.
This ranking determines the problem number and positive number for
each finding. The most severe problem is n01. The most
positive positive finding is p01.
Rename each video clip to include its finding number:
nXX-tpY-keywords.mp4
pXX-tpY-keywords.mp4
where XX is the number (ranking) of the
corresponding positive or negative finding.
For example:
n01-tp4-unclear-when-delivery-free.mp4
p01-tp3-reg-clear.mp4
For the top three positives and top five problems, list (and link) all of the available video clips in the mini-table. Choose the best available video clip to illustrate the finding as a video figure.
In the list of all positives and list of all problems, list (and link) all of the available video clips for each finding.
The thinking aloud test report should contain:
Executive summary. A summary of the test and the most important findings, written for your client's management. In real life, a manager would typically not have time to read the whole report, but would read only the executive summary to find out how the test was run and what the main findings were.
The description of the test procedure, thinking aloud methodology, and the scope of the test should take up no more than 25% of the executive summary. Most of the executive summary should summarise the main findings.
Test Users. Assumptions about user population and their typical tasks, selection process for test users. Test person data (age, occupation, experience, etc.) from background questionnaires in tabular form. Replace all users' real names with fictitious first name aliases in the main TA Report.
Task Completion. Overview of task completion rates.
First Impressions. Summary of the first impressions of the users after Task 1.
Analysis of main positive findings. For each positive finding:
List of All Positive Findings.
Analysis of main negative findings (problems) with recommendations. For each problem:
List of All Problems Found.
Interviews. Summary of the comments and suggestions made by users during the post-test interviews.
Feedback Questionnaires. Summary of the responses given by users in the post-test questionnaire.
User Materials. The filled-out background questionnaires and feedback questionnaires must be scanned to PDF and included as follows:
bq/
tp1-bq.pdf
tp2-bq.pdf
...
fq/
tp1-fq.pdf
tp2-fq.pdf
...
Note: These should be good quality scans. Photos are not acceptable!
For the TA Report:
Follow any instructions and guidance embedded in the skeleton {the text in curly brackets}.
Remove any instructions and guidance {the text in curly brackets} from the file before you hand it in.
All text files (HTML, CSS, etc.) must be encoded in UTF-8 format.
Make sure that when you edit such a file, when you save it, it is still in UTF-8 format. Check the settings in whatever plain text editor you are using.
Write straightforward, simple, valid XHTML5 using a plain text editor or an IDE like Visual Studio Code (single file, no frames, no Javascript, no export from Word, no conversion from LaTeX).
Do not prettify or change the structure of the HTML code. Use two
spaces for indentation. Do not use any Tab characters. Leave
the <section> structure
and <section> ids intact. Do not make any changes
to the CSS file or add any <style> elements to the
HTML.
To ensure that your HTML is well-formed, make sure that it is valid XHTML5. See my Guide to Validating XHTML5.
If you are not familiar with XHTML5, consult Chapters 10 and 11 of my INM 2014 lecture notes.
All video clips and images referenced in the report must be handed in as local copies (referenced by relative links), so that the report is self-contained. Do not link to such assets remotely.
Video clips for the top three positives and top five problems should
be given space in the report using the HTML5 video
element, as shown in the skeleton template.
By default, the HTML5 video element initially displays
the first frame of video. For our video clips, it is often more
appropriate to display a frame from somewhere within the video clip as
the initial image. This can be achieved by extracting a still frame
(in JPEG format) which better illustrates the finding, for
example n01-tp4-unclear-when-delivery-free.jpg for a
video clip named n01-tp4-unclear-when-delivery-free.mp4,
and specifying it in the poster attribute of
the video element.
Remember, all the video clips which are included in the TA Report must be in MP4 (H.264 video and AAC audio) format, in landscape mode, no longer than 20 seconds duration, no larger than 10 MB, and no larger than FullHD resolution (1920×1080 pixels) .
Shorten or transcode any of the videos to be handed in with the report which exceed these limitations.
And, I ask (but cannot require) that you place your TA Report under a CC BY 4.0 licence. To do so, leave the following statement in place at the bottom of the report:
This work is placed under a Creative Commons Attribution 4.0 International (CC BY 4.0) licence.
If you do not wish to do so, then remove the statement.
Create a slide deck for a 15-minute group presentation of your TA Report to your client (tutor).
Use slides (screenful by screenful) with bulleted lists (not full sentences).
Make a separate presentation. Do not simply open your report in a web browser and project that.
Your slides should be in English and you should present in English.
Use PowerPoint. If you do not have PowerPoint, try LibreOffice Impress
as an alternative. Save the file in PowerPoint .pptx
format. Do not use Google Slides!
Students at Graz University of Technology have free access to
Microsoft Office 365 by going
to portal.office.com
and logging in with their university account.
In Google Slides, it is not possible to embed video within the slide deck file, only to link it, which breaks if a video file moves or is deleted, or if you are offline.
You may have to install a video codec pack (such as the K-Lite Codec Pack for Windows or the GStreamer plugins for Linux), to play back video clips from within PowerPoint or Impress.
For the slide deck:
.pptx file
and the presentation is completely self-contained.
Create a title slide with the following information:
And, I ask (but cannot require) that you place your slide deck under a CC BY 4.0 licence. To do so, include the following statement in a smaller font at the bottom of the title slide:
This work is placed under a Creative Commons Attribution 4.0 International (CC BY 4.0) licence.
Address (at least) the following talking points within the slide deck:
Name the
presentation gT-GG-ta-slides.pptx,
where T-GG is the number
of your tutor and group.
For
example, g1-01-ta-slides.pptx for Group G1-01.
Make a directory gT-GG-ta for your
thinking aloud test report, where T-GG
is the number of your tutor and group. For
example, g1-01-ta for Group 1-01.
Your main file should be named ta.html.
Copy over the file report.css unchanged.
Copy over the various user-facing materials. If you are testing in
German with German-speaking test users, use the user-facing
materials (background questionnaire, feedback questionaire, etc.) in
German with the suffix -de.
Place any images into a subdirectory called images.
Create a subdirectory presentation for your presentation
slides.
Place the video clips and poster images into a subdirectory
called videos.
When naming your files and directories, use only lower case letters, digits, and hyphens, from the 7-bit ASCII character set.
Your directory structure should look something like this:
g1-01-ta/
background.html
checklist.html
consent-ta.pdf
external-tasks-ta.pdf
feedback.html
interview.html
orient.html
report.css
ta.html
bq/
tp1-bq.pdf
tp2-bq.pdf
...
fq/
tp1-fq.pdf
tp2-fq.pdf
...
images/
browser-version.png
testroom01.jpg
testroom02.jpg
...
presentation/
g1-01-ta-slides.pptx
videos/
n01-tp4-unclear-when-delivery-free.jpg
n01-tp4-unclear-when-delivery-free.mp4
n02-tp2-shipping.jpg
n02-tp2-shipping.mp4
...
p01-tp3-reg-clear.jpg
p01-tp3-reg-clear.mp4
...
You may have some -de suffixes, if you are using the
German versions of user-facing materials.
Tidy up your directories. Do not leave junk files, backup files, etc. lying around.
Make a zip file of your hand-in directory, including the directory itself (not just the files inside).
Name the zip file gT-GG-ta.zip,
where T-GG is the number of your tutor
and group. For example, g1-01-ta.zip for Group 1-01.
The maximum size of your zip file is 500 MB (500,000,000 bytes). Check the size before uploading.
If your files are too big, you may have to delete something or reduce the size of something. Contact your tutor if you are unsure what to remove or make smaller.
Upload your zip file to TeachCenter before the submission deadline.
This exercise is a group exercise. The group makes a single submission (one zip file) as a group.
If you make any changes to your submission after the deadline, your submission will be flagged with the new timestamp and will be considered to be a late submission with the corresponding points deduction.
Submissions will cease to be accepted 48 hours after the deadline.
At the meeting, you must present the same version of your work which was handed in (uploaded to TeachCenter).
Note that, under Ex3b, you will also hand in a USB stick at Meeting M3, containing the full-length videos from the tests as well as other personally identifiable test user data.