First of all, read the sections in the lecture notes on usability testing.
One more time. I am assuming you have been attending the lectures and know all about usability testing and thinking aloud tests. If you have not, you must read the lecture notes and catch up.
Preparation:
For a site available in both English and German, choose one of the two languages to test. All of your tests must be run in that language:
If you are testing an English-language web site, choose test users who speak English, and run the test in English. User-facing materials should be in English.
If you are testing a German-language web site, choose test users who speak German, and run the test in German. User-facing materials should be in German.
Consider the representative user profile(s) for your web site. Often there are several different user groups. In a TA test with five users, only one of these possible user groups should be chosen for testing, and the five test users should be recruited from that group.
Suggest the user group to test, which you think can best be simulated by the test users which you (and your co-groups) can get hold of, i.e. presumably fellow students of various kinds.
In real life, you would probably run separate TA tests for each user group.
We will notify you of which user group and which tasks you should actually use for your TA tests shortly after meeting M2, once we have received the suggestions from all the groups testing that web site.
Organise five potential test users from the chosen user group, one user for the pilot test and four users for the real test. Assign (sensible) first name aliases to your test users, different from their real first names.
The pilot user should come from the middle of the user group (not the extremes).
The four users for the real test should span the range of the user group (as far as is possible).
When selecting test users in the real world, you typically ask them some questions from a screening questionnaire to make sure they fit the profile of the user group you are testing.
For these practicals, make sure that your test users meet the following criteria:
At least 18 years of age.
Native or proficient speaker of German (if you are running your tests in German) or English (if you are running your tests in English).
Experienced in using both computers and the web.
Not serving as a test user for another HCI group this semester.
Not currently taking HCI this semester.
Organise a quiet room with a PC (= desktop or laptop, running Windows, MacOS, or Unix) and an internet connection.
Plan to leave one full day between pilot test and real tests.
Plan to run the tests with one of Chrome, Firefox, or Safari on a PC. All test users must use the same device and browser!
For these tests, do not use an ad blocker.
Leave it up to each test user to decide what they do with cookies.
Plan to use session capture software and a webcam on the PC to capture each test session. See my Guide To Session Capture. For the thinking aloud tests, you should plan to capture usercam video as well, in addition to audio and the screen.
The session recording should be at most at FullHD resolution (1920×1080 pixels) in landscape orientation. If your device has higher resolution than FullHD, please plan to adjust the settings or make the browser window smaller. If your device has lower resolution than FullHD, record at its maximum resolution. The standard browser window and GUI should be included in the recording. Other things outside the browser window should not be included in the recording, except possibly the webcam with the user's face.
If the native resolution of your device is too high in one or both directions, and you cannot adjust the settings or make the browser window smaller, you will have to record at higher pixel resolution and later transcode down uniformly to at most FullHD. Do not distort the video and do not create black or empty strips to either side (or top and bottom). See my Guide to Video Transcoding. Unfortunately, transcoding always results in a loss of quality.
For the session recordings:
Do not use recording software which leaves behind a watermark.
Plan to turn on recording of the mouse pointer (possibly even make it slightly larger), but turn off any mouse trails.
Plan to record in MP4 format with H.264 video and AAC audio if possible. If your device/software can record in that format natively, that is perfect.
Otherwise, you will have to convert/transcode your video clips to MP4 (with H.264 video and AAC audio) format later on. Unfortunately, transcoding always results in a loss of quality.
Use the following recommended video settings:
| Container: | MP4 |
| Output Video Resolution: | 1920×1080 (FullHD) |
| Frame Rate: | 20 |
| Codec: | H.264 |
| Rate Control: | VBR (Variable Bit Rate) |
| Bit Rate: | 5000 Kbps (= 5 Mbps) |
To capture user audio in the session video, use a built-in or external webcam microphone or an external microphone plugged into the device.
Use the following recommended audio settings:
| Channels: | Stereo |
| Codec: | AAC (= mp4a) |
| Sample Rate: | 44100 Hz |
| Bit Rate: | 160 Kbps |
| Bits per Sample: | 32 |
To capture the user's face in the session video, use a built-in or external webcam and place the picture, say, in the bottom right-hand or bottom left-hand corner of the screen, depending on what works best for your web site.
If your screen is large enough, you could also make the browser window a little narrower, and place the webcam in the margin to the right of the browser, at the bottom. This way, it will not overlap any content of the browser window.
Later, for the full TA Report, you will have to create short video clips illustrating the findings. Plan how you will obscure the user's face in these video clips by blurring, pixelating or blacking out. See my Guide to Face Blurring.
In addition to session capture software on the test device, organise external video recording equipment (tripod, video camera, microphone, and mirror). To be clear: You must use a tripod-mounted external video camera and a mirror in any case, in addition to session capture on the test device.
It is much more convenient in terms of scheduling for you, your test users, and us, if you can arrange your own external video recording equipment. If you cannot, a limited number of sets of video equipment will be available for loan, see the kit and room slots.
For the external video recordings:
Our preferred video format is MP4 (H.264 video with AAC audio) at 720p (1280×720).
This provides good quality, but does not have the larger file sizes of FullHD.
Do not record at less than 1280×720 or higher than 1920×1080 (FullHD) resolution.
Record in landscape mode (the video is wider than it is higher), not portrait mode.
If your external video camera does not record in MP4 with the above
codecs, we will try to play any of the following formats: MPEG-4 with
different video or audio codecs (.mp4), MPEG-1 or MPEG-2
(.mpg), QuickTime (.mov), Matroska
(.mkv), and WebM (.webm). Please do not use
any obscure or proprietary video or audio codecs.
If we have difficulties playing your external video files, you may have to convert your videos into a different format and hand them in again.
Plan to use an external microphone to capture audio on the external video camera:
Use headphones to monitor the audio as it is being recorded. Make sure the audio of the test user is loud and clear.
You can try using the camera's built-in microphone, but since it will be located behind the test user, the audio of the test user might be too quiet (and the microphone may pick up other annoying background sounds too).
Turn off any data fields (date, time, mode, etc.) in the camera which would otherwise overlay the video recording.
Make a test recording to check how the camera works, how to select a format and resolution, that the external microphone works, the audio levels are high enough, and the type and size of video files produced.
For the thinking aloud tests in this course, the external video camera must be used to record the entire test procedure: the user being greeted, being asked the questions on the background questionnaire, receiving training, performing the tasks, being interviewed after the final task, and filling out the feedback questionnaire.
Use the same test environment (room, internet connection, device, browser, video equipment) for all five test users.
Unless, of course, you discover a problem and have to swap a piece of equipment.
It is OK for members of the test team to swap roles between tests (e.g. the facilitator for one test can be the video operator for the next test).
Create suggestions for four tasks (Task 2 to Task 5 below) for users from your chosen user group to perform on your web site. If the tasks have a natural ordering, say a user has to register before having full access to the site, then arrange the tasks in their natural order. Otherwise, arrange the tasks in increasing order of difficulty:
For each task specify:
For registration tasks, users may have to enter personal details, such as a name and address. They can either use their own real identity, if they are comfortable with that, or they can use an obviously fictitious identity (such as Max Mustermann, Musterplatz 1, 8010 Graz), as they please. You should prepare a fictitious identity for each test user in advance, to hand to the user when the time comes. This may also involve preparing free email addresses in advance, one for each test user to use.
For an ordering or submission process, the user should go through the process, up to the point where they are about to complete the order or submission (when you should stop them!). They should not actually go through with any order or submission, unless of course they are using their real identity and they really do want to do it.
From the user groups and tasks suggested by all the groups for your web site, your tutor and I will select the user group and tasks to be used in your test. You will be notified through the course chat room on Tue 30 Apr 2024. That means you cannot start with your pilot test until after that date.
You must use the user group and tasks that we specify. These will not necessarily be the same as the ones which your group suggested.
In a real world situation, you would select test users representative of the user group to be tested (and you would know the user group in advance). For these practicals, we are somewhat restricted in our choice of test users, given that we have no budget available and are dependent on the good will of your friends and colleagues. Furthermore, you will have already organised a number of potential test users and listed them in your TA Plan before the chosen user group is announced.
We will try to select as broad a user group as possible (and as studenty a user group as possible). When the user group for your web site is announced, check whether your suggested test users broadly match the chosen user group. If some do not, do your best to find replacements (do the best you can). If, in the end, some of your test users do not match the target user group exactly, so be it.
Explain how you will deal with data protection. You should plan to delete any personal data of the users (scans of consent forms, full video recordings, mapping of real names to ids and aliases) one year afterwards.
Backups must also be deleted: back up your video recordings in one or two well-defined locations, so you know where to find and delete them. Be careful with cloud backups, turn off snychronisation for the personal data. If you are recording the external video on a device with cloud sync, deactivate it for these recordings.
Write a short plan for the thinking aloud test, by following the skeleton provided in the materials.
In addition:
Adapt the orientation script for your test (first name of facilitator, first names of test team, etc.).
Extend the background questionnaire with three domain-specific questions applicable to your test web site.
Adapt the external task slips to match your suggested tasks.
Extend the standard questions in the interview to at least four questions (two are already suggested).
Extend the feedback questionnaire with two domain-specific questions applicable to your test web site.
For the TA Plan:
Use the skeleton provided.
Follow any instructions and guidance embedded in the skeleton {the text in curly brackets}.
Remove any instructions and guidance {the text in curly brackets} from the file before you hand it in.
All text files (HTML, CSS, log files, etc.) must be encoded in UTF-8 format.
Make sure that when you edit such a file, when you save it, it is still in UTF-8 format. Check the settings in whatever plain text editor you are using.
Write straightforward, simple, valid XHTML5 using a plain
text editor
(single file, no frames, no Javascript,
no specialised HTML editor like Dreamweaver, no export from Word, no
conversion from LaTeX). Do not change the <section>
structure or the <section> ids. Do not make any
changes to the CSS file or add any <style> elements
to the HTML. Make sure your HTML is
valid XHTML5.
If you are not familiar with XHTML5, consult Chapters 10 and 11 of my INM 2014 lecture notes.
Prepare a 5-minute slide presentation of your TA Plan to show to your client at the client meeting (M2):
Make a separate presentation. Do not simply open your TA Plan in a web browser and project that.
You may prepare your slide presentation in Powerpoint, Keynote, HTML, or PDF as long as you can export it (without loss of fidelity) to PDF.
Export your slide presentation to PDF and name
it taplan-slides.pdf
Open your PDF slide presentation with a PDF Viewer and make sure it can be used to give a presentation.
Make a directory gT-GG-taplan for your
thinking aloud test plan, where T-GG is
the number of your tutor and group. For
example, g1-05-taplan for Group 1-05.
Copy the skeleton plan and other files into this directory.
Your main file should be named taplan.html.
For materials which your test users will encounter (background
questionnaire, consent form, feedback questionaire, etc.), choose the
appropriate language version (with suffix -de for German
content).
Create a subdirectory presentation for your
presentation slides.
When naming your files and directories, use only lower case letters, digits, and hyphens, from the 7-bit ASCII character set.
Do not use upper case letters, spaces, underscores, umlauts, or special characters. Only use dot (.) to designate a file's extension, nowhere else in a file name and nowhere in a folder name.
Your directory structure should look like something this:
g1-05-taplan/
background.html
checklist.html
consent-ta.pdf
do-not-disturb.pdf
external-tasks-ta.pdf
feedback.html
orient.html
report.css
taplan.html
...
presentation/
taplan-slides.pdf
You may have some -de suffixes, if you are using the
German versions of user-facing materials.
Tidy up your directory. Do not leave junk files, backup files, etc. lying around.
Your directory must occupy 20 MB (20,000,000 bytes) or less. Check the size before uploading.
If your files are too big, you may have to delete something or reduce the size of something. Sometimes, you can significantly reduce the size of your slides PDF, without significantly affecting quality. Contact your tutor if you are unsure what to remove or make smaller.
Make a zip file of your directory and keep it somewhere else as a backup.
Do not upload the zip file to Sapphire.
Submit your TA Plan before the deadline to the Sapphire online submission system:
https://sapphire.isds.tugraz.at/
See my Guide to Using Sapphire.
Ex 3 is a group exercise. The group makes a single submission as a group.
Every member of the group has access permissions to create and modify the group's submission, but it is best to designate a single person to do the uploading or any modifications, in order to avoid potential conflicts with other group members making changes in parallel.
Use the following approach to upload your files:
Go to Exercises and select “Ex 3: TA Plan”.
Create a submission for Ex 3.
Click on Upload.
Drag your local submission
folder gT-GG-taplan/ (yes, the folder,
for example g1-05-taplan/, not all of the individual
files) and drop it into the upload area.
Do not upload your submission as a zip file.
Watch while the files upload (any sub-folder structure will be created appropriately).
Check your submission folder to make sure everything is there and looks right.
If you make any changes (including deleting files) to your submission after the deadline, your submission will be flagged with the new timestamp and will be considered to be a late submission with the corresponding points deduction.
Submissions will cease to be accepted 48 hours after the deadline.
At the client meeting, you must present the same version of your work which was handed in (uploaded to Sapphire).