Test one test user, taken from the middle of the target user group.
Leave at least one full day between pilot test and real tests, to make any changes which might be necessary.
Test 4 test users, spanning the target user group.
Analyse the test videos and notes.
For each test user:
Greet the user and explain the purpose of the test using the orientation script provided (adapted to your web site).
Ask the user the questions on the background questionnaire. The facilitator should hold the pen and write the answers in the form, not the user.
Make sure the TPID of the test user (of the
form HCI-SSyyyy-Gx-xx-TPn, for
example HCI-SS2023-G1-01-TP1) is written on the top right
of the consent form.
Explain how you will deal with data protection: that you will delete any personal data of the users (scans of consent forms, full video recordings, mapping of real names to ids and aliases) one year afterwards.
Ask the user to read and sign the consent form provided. Answer any questions they may have.
Provide the user with a copy of the form, or let them take a photo of the form if they wish.
Go to uncut.at
and find out which films are on tonight at the Royal English
Cinema, whilst thinking aloud yourself and explaining everything you do.
Show the user a short video clip of a previous user who is giving lots of good feedback (on a different interface). You may use this video clip (13,244,026 bytes), if you wish.
Reset the browser to a fresh state for each new user:
Make sure all add-ons and extensions are disabled.
Delete all cookies.
Delete all temporary files (clear the cache).
Delete any previous browsing history.
Set the page being displayed to google.com
(or google.at for German-speaking test users).
Run the test using the tasks provided:
The facilitator should read out the task slip and then place on the table next to the computer.
The scribe should take notes on paper or in a text file or spreadsheet on a laptop.
Interview the user, beginning with the open question: "So, how was it?". Then ask your pre-prepared interview questions.
After the interview, ask the user to fill out the feedback questionnaire provided. The user should fill out the feedback questionnaire themselves (the facilitator hands them the pen).
Note:
As outlined in the TA Plan, run the tests on a PC (= desktop or laptop, running Windows, MacOS, or Unix) with Chrome, Firefox, or Safari. All test users must use the same device and browser!
For these tests, do not use an ad blocker.
Leave it up to each test user to decide what they do with cookies.
You must record the entire test procedure, from the moment the user enters the room to the moment the user leaves the room, with an external tripod-mounted video camera. This applies both to the pilot test and the real tests! See my Guide to External Video Recording.
If you greet the user at a separate table, set up the external camera to initially point at the table (filming the introduction, background questionnaire, and consent form), then as the user moves over to the test PC, swing the camera around to shoot over the shoulder of the user for the actual test session.
Use a mirror next to the monitor or screen to capture the user's facial expressions.
When filming the test session on the external camera, set up the field of view (recording area) to film over the shoulder of the user, so that the screen, keyboard, and the user's head in the mirror are framed in the picture.
Avoid excessive panning and zooming of the external video camera. You must use a tripod. Do not hand-hold the camera or move it around unnecessarily.
Use an external microphone to capture audio on the external video camera.
Use headphones to monitor the audio as it is being recorded. Make sure the audio of the test user is loud and clear.
You can try using the camera's built-in microphone, but since it will be located behind the test user, the audio of the test user will probably be too quiet (and the microphone may pick up other annoying background sounds too).
Turn off any data fields (date, time, mode, etc.) in the camera which would otherwise overlay the video recording.
For the external video recordings:
Our preferred video format is MP4 (H.264 video with AAC audio) at 720p (1280×720).
This provides good quality, but does not have the large file sizes of FullHD.
Do not record at less than 800×600 or higher than 1920×1080 (FullHD) resolution.
Record in landscape mode (the video is wider than it is higher), not portrait mode.
If your external video camera does not record in MP4 with the above
codecs, we will try to play any of the following formats: MPEG-4 with
other video or audio codecs (.mp4), MPEG-1 or MPEG-2
(.mpg), QuickTime (.mov), Matroska
(.mkv), and WebM (.webm). Please do not use
any obscure or proprietary video or audio codecs.
If we have difficulties playing your external video files, you may have to convert your videos into a different format and hand them in again.
You must also capture each test user's session on the device, including the screen, the user's voice, and the user's face. See my Guide to Session Capture.
You must both capture the test session on the device (session capture) and also record the entire test using an external video camera.
Record on a PC in landscape orientation. The screen recording should be at most at FullHD resolution (1920×1080. If your PC has higher resolution than FullHD, please plan to adjust the settings or make the browser window smaller. If your device has lower resolution than FullHD, record at its maximum resolution. The standard browser window and GUI should be included in the recording. Other things outside the browser window should not be included in the recording.
If the native resolution of your device is too high in one or both directions, and you cannot adjust the settings or make the browser window smaller, you will have to record at higher pixel resolution and later transcode down uniformly to at most FullHD. Do not distort the video and do not create black or empty strips to either side (or top and bottom). See my Guide to Video Transcoding. Unfortunately, transcoding always results in a loss of quality.
Do not use recording software which leaves behind a watermark. Turn on recording of the mouse pointer (even make it slightly larger), but turn off any mouse trails.
Record in MP4 format with H.264 video and AAC audio if possible. If your device/software can record in that format natively, that is perfect.
Otherwise, you will have to convert/transcode your video clips to MP4 (with H.264 video and AAC audio) format later on. Unfortunately, transcoding always results in a loss of quality.
Use the following recommended video settings:
| Container: | MP4 |
| Output Video Resolution: | 1920×1080 (FullHD) [or 1080×1920] or lower |
| Frame Rate: | 20 |
| Codec: | H.264 |
| Rate Control: | VBR (Variable Bit Rate) |
| Bit Rate: | 5000 Kbps (= 5 Mbps) |
To capture user audio in the session video, use a built-in or external webcam microphone or an external microphone plugged into the device.
Use the following recommended audio settings:
| Channels: | Stereo |
| Codec: | AAC (= mp4a) |
| Sample Rate: | 44100 Hz |
| Bit Rate: | 160 Kbps |
| Bits per Sample: | 32 |
To capture the user's face in the session video, use a built-in or external webcam and place the picture, say, in the bottom right-hand or bottom left-hand corner of the screen, depending on what works best for your web site.
If your screen is large enough, you could also make the browser window a little narrower, and place the webcam in the margin to the right of the browser, at the bottom. This way, it will not overlap any content of the browser window.
You will need to watch each session capture video, including the pilot test, and manually transcribe the events into a chronological protocol for the TA report.
The timestamps in the report refer to running time in hours minutes and seconds (hh:mm:ss) from the start of the session capture video. They allow the reader to easily locate (fast forward or jump to) events of interest on the videos.
Enter the information from the backgound and feedback
questionnaires into the spreadsheet data-ta.xlsx.
Enter the task success metrics into the spreadsheet.
Analyse the session capture videos and collect a list of positive findings and negative findings (problems) in the spreadsheet. Note the timestamp where the finding started.
For each positive or negative finding, determine which test users experienced which findings. For each finding, create one video clip for each test user who experienced this finding to illustrate what happened. The video clips should be in MP4 (H.264 video and AAC audio) format and be no larger than FullHD (1920×1080) resolution; transcode if necessary. Each video clip should be no longer than 20 seconds in duration and no larger than 10 MB in size.
The short video clips illustrating findings are considered to be part of the report. They should not contain identifiable faces of the test users. Obscure the user's face on the video clips by blurring, pixelating or blacking out. See my Guide to Face Blurring.
Initially name the video clips according to the following naming scheme:
n-tpY-keywords.mp4
p-tpY-keywords.mp4
where n indicates a negative finding (problem)
and p indicates a positive finding, Y is
replaced by the number of the test user where the finding was observed
(e.g. tp1), and keywords comprises one or
more words describing the finding. If you need to, you can use upto
four keywords separated by hyphens. Use only lower case letters,
digits, and hyphens from the 7-bit ASCII character set.
For example:
n-tp4-free-deliv.mp4
p-tp3-reg.mp4
The finding number will be inserted later, once the severity (or
positivity) of all findings has been determined and the findings have
been sorted and ranked (e.g. n01-tp4-free-deliv.mp4).
We would expect each group to find between 10 and 30 problems (over all test users), depending on the quality of the web site you are testing. We would expect each group to find at least 5 positives (over all test users).
Individually rate the positive findings by positivity and the problems by severity (working alone), then form a group average (as for the heuristic evaluation). Enter these into the spreadsheet and sort the tables in descending order.
This determines the problem number and positive number for each
finding. The most severe problem is n01. The most
positive positive finding is p01.
Rename each video clip to include its finding number:
nXX-tpY-keywords.mp4
pXX-tpY-keywords.mp4
where XX is the number of the positive or negative
finding.
For example:
n01-tp4-free-deliv.mp4
p01-tp3-reg.mp4
For the top three positives and top five problems, list (and link) all of the available video clips in the mini-table. Choose the best available video clip to illustrate the finding as a video figure.
In the list of all positives and list of all problems, list (and link) all of the available video clips for each finding.
When using the HTML5 video element, the first frame of
the video is displayed by default (before the video is played). For
most video clips, the first frame of the video does not
adequately illustrate the finding. Instead, extract a still frame (in
JPEG format) which better illustrates the finding, to act as a poster
image for the video clip. For example:
n01-tp4-free-deliv.jpg
p01-tp3-reg.jpg
Specify the name of the poster image in the
poster attribute of the video element.
The thinking aloud test report should contain:
Executive summary. A summary of the test and the most important findings, written for your client's management. In real life, a manager would typically not have time to read the whole report, but would read only the executive summary to find out how the test was run and what the main findings were.
The description of the test procedure, thinking aloud methodology, and the scope of the test should take up no more than 25% of the executive summary. Most of the executive summary should summarise the main findings.
Test Users. Assumptions about user population and their typical tasks, selection process for test users. Test person data (age, occupation, experience, etc.) from background questionnaires in tabular form.
Replace all users' real names with fictitious first names in the main TA Report.
Analysis of main positive findings. For each positive finding:
Analysis of main negative findings (problems) with recommendations. For each problem:
Interviews. Summary of the comments and suggestions made by users during the post-test interviews.
Feedback questionnaire. Summary of the responses given by users in the post-test questionnaire.
Transcript of session capture videos. As an appendix a chronological table of events, comments, problems. Timestamps refer to the running time from the start of the video in hours, minutes, and seconds (hh:mm:ss), i.e. how far a viewer would have to jump forward to find a particular event.
The filled-out user materials: background questionnaires and feedback questionnaires must be scanned to PDF and included as follows:
bq/
tp1-bq.pdf
tp2-bq.pdf
...
fq/
tp1-fq.pdf
tp2-fq.pdf
...
Note: These should be scans in reasonably good quality. Photos are not acceptable!
For the report:
Follow any instructions and guidance embedded in the skeleton {the text in curly brackets}.
Remove any instructions and guidance {the text in curly brackets} from the file before you hand it in.
All text files (HTML, CSS, etc.) must be encoded in UTF-8 format.
Make sure that when you edit such a file, when you save it, it is still in UTF-8 format. Check the settings in whatever plain text editor you are using.
Write straightforward, simple, valid XHTML5 using a plain
text editor
(single file, no frames, no Javascript,
no specialised HTML editor like Dreamweaver, no export from Word, no
conversion from LaTeX). Do not change the <section>
structure or the <section> ids. Do not make any
changes to the CSS file or add any <style> elements
to the HTML. Make sure your HTML is
valid XHTML5.
If you are not familiar with XHTML5, consult Chapters 10 and 11 of my INM 2014 lecture notes.
All video clips and images referenced in the report must be handed in as local copies (referenced by relative links), so that the report is self-contained. Do not link to such assets remotely.
Make a directory gT-XX-ta for your thinking aloud
test report, where T-XX is the number of your tutor and
group. For example, g1-05-ta for Group 1-05.
Copy the skeleton plan and other files into this directory.
Your main file should be named ta.html.
Your spreadsheet should be named data-ta.xlsx.
If you are testing in German with German-speaking test users, choose
the user-facing materials (background questionnaire, feedback
questionaire, etc.) in German with the suffix -de.
Place any images into a subdirectory called images.
Place the video clips and poster images into a subdirectory
called videos.
Prepare a 15-minute slide presentation of your TA Report to show to your client at the client meeting (M3):
Make a separate presentation. Do not simply open your TA Report in a web browser and project that.
Most of your allotted presentation time should be used to show slides. Do not show more than three short video clips.
You may prepare your slide presentation in Powerpoint, Keynote, HTML, or PDF as long as you can export it (without loss of fidelity) to PDF.
Export your slide presentation to PDF, name
it ta-slides.pdf and place it in a subdirectory
called presentation.
Open your PDF slide presentation with a PDF Viewer and make sure it can be used to give a presentation.
When naming your files and directories:
Your directory structure should look something like this:
gT-XX-ta/
background.html
checklist.html
consent-ta.pdf
data-ta.xlsx
do-not-disturb.pdf
external-tasks-ta.pdf
feedback.html
orient.html
report.css
ta.html
bq/
tp1-bq.pdf
tp2-bq.pdf
...
fq/
tp1-fq.pdf
tp2-fq.pdf
...
images/
browser-version.png
testroom01.jpg
testroom02.jpg
...
presentation/
ta-slides.pdf
videos/
n01-tp4-free-deliv.jpg
n01-tp4-free-deliv.mp4
n02-tp2-shipping.jpg
n02-tp2-shipping.mp4
...
p01-tp1-clear-reg.jpg
p01-tp1-clear-reg.mp4
...
You may have some -de suffixes, if you are using the
German versions of user-facing materials.
Tidy up your directories. Do not leave junk files, backup files, etc. lying around.
Your directory must occupy 350 MB (350,000,000 bytes) or less. Check the size before uploading.
If your files are too big, you may have to delete something or reduce the size of something. Sometimes, you can significantly reduce the size of your slides PDF, without significantly affecting quality. If your video clips are too large, you may be able to shorten them or transcode them using higher compression. Contact your tutor if you are unsure what to remove or make smaller.
Make a zip file of your directory and keep it elsewhere as a backup (do not upload it).
Submit your TA Report before the deadline to the Sapphire online submission system:
https://sapphire.isds.tugraz.at/
See my Guide to Using Sapphire.
Ex 4a is a group exercise. The group makes a single submission as a group.
Every member of the group has access permissions to create and modify the group's submission, but it is best to designate a single person to do the uploading or any modifications, in order to avoid potential conflicts with other group members making changes in parallel.
Use the following approach to upload your files:
gT-XX-ta/ (yes, the
folder, for example g1-05-ta/, not all of the
individual files) and drop it into the upload area.
Submissions will cease to be accepted 48 hours after the deadline.
At the client meeting, you must present the same version of your work which was handed in (uploaded to Sapphire).
Note that for Exercise 4b, you will also hand in a USB stick containing the full-length videos from the tests.