Thinking Aloud Test Report

3.1 Test Methodology

{Describe what a TA Test is and how it is done. Write between 300 and 500 of your own words. Replace my sample text below with your own.}

A thinking aloud test is ... ... can be found in Keith Andrews' course notes [And2023]. The thinking aloud method is described in detail by Barnum [Bar2020].

{Cite at least two more references of your own in this section and include them in the References section below. Use only indirect quotations (paraphrasing), no direct quotations. You may remove or retain my sample reference citations as you wish.}

3.2 User Profiles

{Describe the kinds of user the site is trying to attract.}

{Group these users into categories according to their characteristics.}

{Describe the goals and typical tasks for each of these user groups.}

{Describe the user group you actually tested, i.e. the one you were notified to use.}

3.3 Test Users

{Describe the selection process for test users, to ensure that they come from the user group chosen for testing.}

{Collect all the data from the background questionnaires, including any domain-specific questions, and enter it into the table below. Use the fictitious first name aliases of your test users.}

Table 1 gives an overview of the test users who participated in the study. First name aliases are used throughout this report to protect their identity.

3.4 Test Environment

{Include one or two photos of the room and test environment (JPEG). Anonymise anyone who is identifiable in the photos by pixelating or blurring their faces.}

The test room is shown in Figure 1.

The environment used for the thinking aloud tests is shown in Table 2.

{Fill out the table with the exact details used at the time of the TA tests.}

{Include a screenshot (PNG) of the browser About window to document the exact browser version used in the test.}

The exact web browser version used for the tests is shown in Figure 2.

{Explain that ad blockers were not used (uninstalled, disabled).}

{Explain that use of cookies was left up to each test user.}

{Explain how faces were blurred or pixelated in the screen recordings.}

{Describe the external video equipment (tripod, video camera, microphone, mirror) which was used to record the entire test.}

3.5 Training

{Describe any interface training given to each user. For example, a special input device, window manager, or software.}

{Describe any training on thinking aloud given to each user. How did the facilitator demonstrate the thinking aloud technique? Did the user practice it?}

3.6 Tasks

{Enter the tasks which were actually used, i.e. the ones you were notified to use.}

{If you tested in German, list the tasks in German and provide English translations.}

The internal task list used by the test team is shown in Table 3.

The task descriptions given to the users during the test are shown in Table 4. The actual task slips are included in Appendix A.5.

3.7 Interview Questions

{If you tested in German, list the questions in German and provide English translations.}

The following questions were asked of each user immediately after the final task. Summaries of the interviews can be found in Section 4.7.

• Opening Question

  {The first question should always be "How was it?", an open question to encourage the user to speak freely.}

  • "How was it?" ["Wie war's?"]

    {Then let the user speak, and encourage them, until they dry up and stop of their own accord.}

• Standard Questions

  {Next, you ask a series of standard questions prepared in advance for every test user. More general questions should come first, more specific questions later. At least four standard questions should have been prepared for this section in the TA Plan.}

  1. "Did anything strike you as particularly good?"
     ["Gab es etwas, dass besonders gut oder positiv aufgefallen ist?"]
  2. "Did anything strike you as particularly bad?"
     ["Gab es etwas, dass besonders schlecht war?"]
  3. "..."

• Individual Questions

  After the standard questions above, any individual questions which arose during the test with each particular test user were asked.

3.8 Feedback Questionnaire

After the interview, the user was asked to fill out the feedback questionnaire given in Appendix A.6. The feedback questionnaire was given on paper. An overview of the results can be seen in Section 4.8.

3.9 Data Collection

{Explain that demographic, behavioural, and attitudinal data collected during the study are associated with a TPid and first name alias. Test users are only referred to by these in this report.}

{Explain that personal data (real name, signed consent form, and the full video recordings) collected from the test persons are kept separately and will will be deleted after one year.}

{Explain that any faces in the video clips used to illustrate findings in the report have been blurred or pixelated.}

4.1 Task Completion

A summary of how many users completed each task and whether any assistance was given is shown in Table 5.

{Task completion is often a binary measure, whereby 0 indicates failure and 1 indicates success. Sometimes, a percentage measure of success is used. Explain which measure you are using.}

4.2 First Impressions

Tables 6a, 6b, and 6c show a summary of the responses of the test users to the questions asked after the first task.

{Summarise in English the responses of the test users to the three questions asked after the first task.}

{Never invent content. If you forgot to ask about the first impressions, say so.}

4.3 Top Three Positive Findings

The three most positive findings according to their average (mean) positivity ratings are described in more detail below. The positivity rating scheme used to rank positive findings is shown in Table 7.

{Describe the top three positive findings which emerged during the test, in descending order of mean positivity (most positive first).}

{One instance from every user who experienced a finding should be extracted as a video clip and included in the mini-table for that finding, each linked to the corresponding video clip. List the corresponding timestamp(s), which refer to elapsed time from the start of the corresponding test user's full session capture video.}

{For each finding, include at least one paragraph of text in the description. Choose the best video clip from those available for the finding to embed as a figure in the description.}

4.4 List of All Positive Findings

{Aggregated list of all positive findings observed in the test, in descending order of mean positivity.}

{For every test user who experienced a positive finding, a video clip should be extracted from the session capture video and its name listed in the table cell (linked to the video clip itself). The timestamp of each video clip in the full session capture video should be listed in the neighbouring cell.}

Table 8 shows a list of all the positive findings which emerged from the test, sorted in descreasing order of average (mean) positivity, i.e. the most positive are at the top of the table. Positivity ratings were assigned by members of the test team, their name codes are shown in Table 9.

4.5 Top Five Problems

The five most serious problems according to their average (mean) severity ratings are described in more detail below. Problem number 1 is the problem (negative finding) with the highest mean severity. The severity rating scheme used to rank problems is shown in Table 10.

{Describe the top five problems (negative findings) which emerged during the test, in descending order of mean severity.}

{One instance from every user who experienced a finding should be extracted as a video clip. In the mini-table, include the names of all the video clip(s) for that finding, linked to the corresponding video clip. List the corresponding timestamp(s), which refer to elapsed time from the start of the corresponding test user's full session capture video.}

{For each problem, choose the best video clip from those available for the problem to embed as a figure. Include at least one paragraph of text describing the problem itself. Include a further paragraph of text describing way(s) the problem could potentially be addressed.}

4.6 List of All Problems Found

{Aggregated list of all problems observed in the test, in descending order of mean severity.}

{For every test user who experienced a problem, a video clip (MP4 H.264 AAC) should be extracted from the session capture video and its name listed in the table cell (linked to the video clip itself). The timestamp of each video clip in the full session capture video should be listed in the neighbouring cell.}

Table 11 shows a list of all the problems observed in the test, sorted in descreasing order of average (mean) severity, i.e. the most severe are at the top of the table. Severity ratings were assigned by members of the test team, their name codes are shown in Table 12.

4.7 Interviews

{For each test user, summarise in English the main points made by the user in the their interview.}

TP1 “Stuart” appreciated the simplicity of the registration process, ...

TP2 “Silvia” stated immediately that the...

TP3 “Sally” had to be encouraged to talk out loud, but was very forthcoming in her interview. She stressed that...

etc.

{Never invent content. If you forgot to do the interviews, say so.}

4.8 Feedback Questionnaires

{A tabular summary of the responses to the feedback questionnaire, with the mean and standard deviation. Make sure to include the extra domain-specific questions.}

{The neutral scale in the original feedback questionnaire is replaced by a weighted scale between 6 (best) and 0 (worst). The users' responses are then entered into the columns on the right.}

{The mean and standard deviation is calculated for each question and entered with 2 decimal places in the column on the far right.}

{So that the reader can see the overall results at a glance, the point on the scale closest to the mean is marked highlighted through the style sheet.}

{If you tested in German, use English versions of the questions and answers in this table.}

Table 13 shows a summary of the ratings given by users in the feedback questionnaire at the end of the test. The neutral scale in the original feedback questionnaire has been mapped to a weighted scale between 6 (best) and 0 (worst). The numbers in bold indicate the (rounded) mean rating.

The original blank questionnaire can be found in Section A.6. Scans of the questionnaires as completed by each test user can be found in Section B.2.

A.1 Checklist

The following checklist was used for the tests: checklist.html.

A.2 Orientation Script

The following orientation script was used for the tests: orient.html.

A.3 Consent Form

The following blank consent form was used in the tests: consent-ta.pdf.

A.4 Background Questionnaire

The following blank background questionnaire was used in the tests: background.html.

A.5 Task Slips

The following task slips were presented to the test users: external-tasks-ta.pdf.

A.6 Feedback Questionnaire

The following blank feedback questionnaire was used in the tests: feedback.html.

A.7 Presentation Slides

The slides used for the presentation are: ta-slides.pdf.

B.1 Completed Background Questionnaires

{For each test user, include a good quality PDF scan (in colour if applicable) of the completed background questionnaire. A photograph is not acceptable!}

For each test user, the completed background questionnaire was scanned as PDF:

• tp1-bq.pdf
• tp2-bq.pdf
• tp3-bq.pdf
• tp4-bq.pdf
• tp5-bq.pdf

B.2 Completed Feedback Questionnaires

{For each test user, include a good quality PDF scan (in colour if applicable) of the completed feedback questionnaire. A photograph is not acceptable!}

For each test user, the completed feedback questionnaire was scanned as PDF:

• tp1-fq.pdf
• tp2-fq.pdf
• tp3-fq.pdf
• tp4-fq.pdf
• tp5-fq.pdf

C.1 User Aliases

The mapping of test users' aliases to their real names is documented in the file users.html.

C.2 Signed Consent Forms

For each test user, the signed consent form was scanned as PDF:

• tp1-consent.pdf
• tp2-consent.pdf
• tp3-consent.pdf
• tp4-consent.pdf
• tp5-consent.pdf

C.3 External Videos

For each test user, the entire test was captured with a tripod-mounted external video camera in the following files:

• TP1 “Stuart”: tp1-ext.mp4.
• TP2 “Silvia”: tp2-ext.mp4.
• ...
• TP5: “John”: tp5-ext.mp4.

C.4 Session Capture Videos

For each test user, the entire test session on the computer was captured in the following files:

• TP1 “Stuart”: tp1.mp4.
• TP2 “Silvia”: tp2.mp4.
• ...
• TP5: “John”: tp5.mp4.