First of all, read the sections in the lecture notes on usability testing.
One more time. I am assuming you have been attending the lectures and know all about usability testing and thinking aloud tests. If you have not, you must read the lecture notes and catch up.
Preparation:
Consider the representative user profile(s) for your web site. Often there are several different user groups. In a TA test with five users, only one of these possible user groups should be chosen for testing, and the five test users should be recruited from that group.
Suggest the user group to test, which you think can best be simulated by the test users which you (and your co-groups) can get hold of, i.e. presumably fellow students of various kinds.
In real life, you would probably run separate TA tests for each user group.
For a site available in both English and German, choose one of the two languages to test.
If you are testing an English-language web site, choose test users who speak English, and run the test in English. User-facing materials should be written in English.
If you are evaluating a German-language web site, choose test users who speak German, and run the test in German. User-facing materials should be written in German.
Organise five test users from the chosen user group, one user for the pilot test and four users for the real test. Assign (sensible) first name aliases to your potential test users.
We will notify you of which user group and which tasks you should actually use for your TA tests shortly after meeting M2, once we have received the suggestions from all the groups testing that web site.
The pilot user should come from the middle of the user group (not the extremes).
The four users for the real test should span the range of the user group (as far as is possible).
When selecting test users in the real world, you typically ask them some questions from a screening questionnaire to make sure they fit the profile of the user group you are testing.
For these practicals, make sure that your test users meet the following criteria:
At least 18 years of age.
Experienced in using both computers and the web.
Not currently be taking HCI this semester.
You cannot be a member of one group and simultaneously be a test user for another group.
Organise a quiet room with a PC (= desktop or laptop, running Windows, MacOS, or Unix) and an internet connection.
Plan to leave one full day between pilot test and real tests.
Plan to run the tests with Chrome, Firefox, or Safari. All test users must use the same device and browser!
For these tests, do not use an ad blocker.
Leave it up to each test user to decide what they do with cookies.
Plan to use session capture software and a webcam on the PC to capture each test session. See my Guide To Session Capture. For the thinking aloud tests, you should capture user video, in addition to audio and the screen.
In addition to session capture software on the test device, organise external video recording equipment (tripod, video camera, microphone, and mirror). To be clear: You must use a tripod-mounted external video camera and a mirror in any case, in addition to session capture on the test device.
It is much more convenient in terms of scheduling for you, your test users, and us, if you can arrange your own external video recording equipment. If you cannot, a limited number of sets of video equipment will be available for loan, see the kit and room slots.
For the external video recordings:
Our preferred video format is MP4 (H.264 video with AAC audio) at 720p (1280×720).
This provides good quality, but does not have the large file sizes of FullHD.
Do not record at less than 800×600 or higher than 1920×1080 (FullHD) resolution.
Record in landscape mode (the video is wider than it is higher), not portrait mode.
If your external video camera does not record in MP4 with the above
codecs, we will try to play any of the following formats: MPEG-4 with
different video or audio codecs (.mp4), MPEG-1 or MPEG-2
(.mpg), QuickTime (.mov), Matroska
(.mkv), and WebM (.webm). Please do not use
any obscure or proprietary video or audio codecs.
If we have difficulties playing your external video files, you may have to convert your videos into a different format and hand them in again.
Make a test recording ahead of time to check how the camera works, how to select a format and resolution, that the external microphone works, the audio levels are high enough, and the type and size of video files produced.
For the thinking aloud tests in this course, the external video camera must be used to record the entire test procedure: the user being greeted, being asked the questions on the background questionnaire, receiving training, performing the tasks, being interviewed after the final task, and filling out the feedback questionnaire.
Use the same test environment (room, internet connection, device, browser, video equipment) for all five test users.
Unless, of course, you discover a problem and have to swap a piece of equipment.
It is OK for members of the test team to swap roles between tests (e.g. the facilitator for one test can be the video operator for the next test).
Create suggestions for four tasks (Task 2 to Task 5 below) for users from your chosen user group to perform on your web site. If the tasks have a natural ordering, say a user has to register before having full access to the site, then arrange the tasks in their natural order. Otherwise, arrange the tasks in increasing order of difficulty:
For each task specify:
For registration tasks, users may have to enter personal details, such as a name and address. They can either use their own real identity, if they are comfortable with that, or they can use an obviously fictitious identity (such as Max Mustermann, Musterplatz 1, 8010 Graz), as they please. You should prepare a fictitious identity for each test user in advance, to hand to the user when the time comes. This may also involve preparing free email addresses in advance, one for each test user to use.
For an ordering or submission process, the user should go through the process, up to the point where they are about to complete the order or submission (when you should stop them!). They should not actually go through with any order or submission, unless of course they are using their real identity and they really do want to do it.
From the user groups and tasks suggested by all the groups for your web site, your tutor and I will select the user group and tasks to be used in your test. You will be notified through the course newsgroup on Tue 09 May 2023. That means you cannot start with your pilot test until after that date.
You must use the user group and tasks that we specify. These will not necessarily be the same as the ones which your group suggested.
In a real world situation, you would select test users representative of the user group to be tested (and you would know the user group in advance). For these practicals, we are somewhat restricted in our choice of test users, given that we have no budget available and are dependent on the good will of your friends and colleagues. Furthermore, you will have already organised a number of potential test users and listed them in your TA Plan before the chosen user group is announced.
We will try to select as broad a user group as possible (and as studenty a user group as possible). When the user group for your web site is announced, check whether your suggested test users broadly match the chosen user group. If some do not, do your best to find replacements (do the best you can). If, in the end, some of your test users do not match the target user group exactly, so be it.
Explain how you will deal with data protection. You should plan to delete any personal data of the users (scans of consent forms, full video recordings, mapping of real names to ids and aliases) one year afterwards.
Backups must also be deleted: back up your video recordings in one or two well-defined locations, so you know where to find and delete them. Be careful with cloud backups, turn off snychronisation for the personal data. If you are recording the external video on a device with cloud sync, deactivate it for these recordings.
Write a short plan for the thinking aloud test, by following the skeleton provided in the materials.
In addition:
Adapt the orientation script for your test (first name of facilitator, first names of test team, etc.).
Extend the background questionnaire with three domain-specific questions applicable to your test web site.
Adapt the external task slips to match your suggested tasks.
Extend the standard questions in the interview to at least four questions (two are already suggested).
Extend the feedback questionnaire with two domain-specific questions applicable to your test web site.
Adapt the example spreadsheet
data-ta.xlsx
for your tests:
For the TA Plan:
Use the skeleton provided.
Follow any instructions and guidance embedded in the skeleton {the text in curly brackets}.
Remove any instructions and guidance {the text in curly brackets} from the file before you hand it in.
All text files (HTML, CSS, log files, etc.) must be encoded in UTF-8 format.
Make sure that when you edit such a file, when you save it, it is still in UTF-8 format. Check the settings in whatever plain text editor you are using.
Write straightforward, simple, valid XHTML5 using a plain
text editor
(single file, no frames, no Javascript,
no specialised HTML editor like Dreamweaver, no export from Word, no
conversion from LaTeX). Do not change the <section>
structure or the <section> ids. Do not make any
changes to the CSS file or add any <style> elements
to the HTML. Make sure your HTML is
valid XHTML5.
If you are not familiar with XHTML5, consult Chapters 10 and 11 of my INM 2014 lecture notes.
Prepare a 5-minute slide presentation of your TA Plan to show to your client at the client meeting (M2):
Make a separate presentation. Do not simply open your TA Plan in a web browser and project that.
You may prepare your slide presentation in Powerpoint, Keynote, HTML, or PDF as long as you can export it (without loss of fidelity) to PDF.
Export your slide presentation to PDF and name
it taplan-slides.pdf
Open your PDF slide presentation with a PDF Viewer and make sure it can be used to give a presentation.
Make a directory gT-XX-taplan for your thinking aloud
test plan, where T-XX is the number of your tutor and
group. For example, g1-05-taplan for Group 1-05.
Copy the skeleton plan and other files into this directory.
Your main file should be named taplan.html.
Your spreadsheet should be named data-ta.xlsx.
For materials which your test users will encounter (background
questionnaire, consent form, feedback questionaire, etc.), choose the
appropriate language version (with suffix -de for German
content).
Create a subdirectory presentation for your
presentation slides.
When naming your files and directories, use only lower case letters, digits, and hyphens, from the 7-bit ASCII character set.
Do not use upper case letters, spaces, underscores, umlauts, or special characters. Only use dot (.) to designate a file's extension, nowhere else in a file name and nowhere in a folder name.
Your directory structure should look like something this:
gT-XX-taplan/
background.html
checklist.html
consent-ta.pdf
data-ta.xlsx
do-not-disturb.pdf
external-tasks-ta.pdf
feedback.html
orient.html
report.css
taplan.html
...
presentation/
taplan-slides.pdf
You may have some -de suffixes, if you are using the
German versions of user-facing materials.
Tidy up your directory. Do not leave junk files, backup files, etc. lying around.
Your directory must occupy 20 MB (20,000,000 bytes) or less. Check the size before uploading.
If your files are too big, you may have to delete something or reduce the size of something. Sometimes, you can significantly reduce the size of your slides PDF, without significantly affecting quality. Contact your tutor if you are unsure what to remove or make smaller.
Make a zip file of your directory and keep it elsewhere as a backup (do not upload it).
Submit your TA Plan before the deadline to the Sapphire online submission system:
https://sapphire.isds.tugraz.at/
See my Guide to Using Sapphire.
Ex 3 is a group exercise. The group makes a single submission as a group.
Every member of the group has access permissions to create and modify the group's submission, but it is best to designate a single person to do the uploading or any modifications, in order to avoid potential conflicts with other group members making changes in parallel.
Use the following approach to upload your files:
gT-XX-taplan/ (yes, the
folder, for example g1-05-taplan/, not all of the
individual files) and drop it into the upload area.
Submissions will cease to be accepted 48 hours after the deadline.
At the client meeting, you must present the same version of your work which was handed in (uploaded to Sapphire).