Assemble the individual log files and associated video clips from each evaluator in your group.
As a group, form a combined (aggregated) list of problems (and a
combined list of positives) using the spreadsheet
helist.xlsx
as a template:
If the same (or very similar) findings are found by two or more evaluators, combine them into one.
You then have mulptiple video clips to illustrate the finding.
If the same (or very similar) findings are found on two or more devices, combine them into one.
You then have mulptiple video clips to illustrate the finding.
List many small related issues (such as 15 individual typos) as one problem with many instances (rather than 15 problems).
The same for many examples of German content not being available in English (or vice versa).
Indicate which problems were found by which evaluator(s).
For each problem, determine whether the problem is general to all browsers and platforms or specific to a particular browser or platform. Or, if it only occurs when cookies are disabled, or when an ad blocker is used. In such cases, indicate this in the “Only When...” column. If a problem is general, leave the corresponding cell empty.
Proceed analogously for the positive findings.
Individually (not working together), assign severity ratings to each problem. Use the 0--4 integer severity scale below:
| Severity | Meaning |
|---|---|
| 4 | Catastophic Problem |
| 3 | Serious Problem |
| 2 | Minor Problem |
| 1 | Cosmetic Problem |
| 0 | Not a Problem |
Individual ratings must be integers, fractional ratings are not allowed.
Individually (not working together), assign positivity ratings to each positive finding. Use the 0--4 integer positivity scale below:
| Positivity | Meaning |
|---|---|
| 4 | Extremely Positive |
| 3 | Major Positive |
| 2 | Minor Positive |
| 1 | Cosmetic Positive |
| 0 | Not a Positive |
Individual ratings must be integers, fractional ratings are not allowed.
Calculate the mean severity rating for each problem to 2 decimal places. Calculate the mean positivity rating for each positive finding to 2 decimal places.
Sort the problems into descending order of average severity (most severe first). Sort the positives into descending order of average positivity (most positive first).
Renumber the problems and positives, so that the finding at the top of each list is number 1. We will call these the ranked finding numbers. The top-ranked positive finding will henceforth be named P01, the second-ranked positive finding P02, etc. The top-ranked negative finding will be named N01, the second-ranked negative finding N02, and so forth.
For each finding, enter the names of all available video clips for that finding into the column “All Available Video Clips” in the spreadsheet.
For each finding, select the best (most representative) available video clip(s) to illustrate the finding.
It often makes sense to include multiple video clips to illustrate one finding. For example, if they show different aspects of the same issue (say, one on mobile and the other on PC). Do not always select every available video clip from every evaluator for every finding. Typically, there will be one to three video clips per finding.
Copy and rename the selected video clips for each finding, so that the ranked finding number is prepended to the file name, according to the following scheme:
nNN-EE-negXX-keywords.mp4
pNN-EE-posXX-keywords.mp4
where n indicates a negative finding (problem)
and p indicates a positive finding,
NN is the ranked finding number,
EE are the lower case initials of the evaluator,
and posXX or negXX is the original numbering
in discovery order. For example:
n01-ka-neg01-links-yellow.mp4
p01-ka-pos01-breadcrumbs-work-well.mp4
where n01-ka-neg01-links-yellow.mp4 is one of the
selected video clips illustrating the highest ranked (most severe)
problem N01 and was found by evaluator KA.
Enter the new names for each selected video clip into the column “Selected Video Clip(s)” in the spreadsheet.
Transfer the sorted lists of problems and of positives into simple XHTML5 tables.
Do not just save the spreadsheet from Excel as HTML, since this creates bloated HTML code, which will not be valid XHTML5.
For example, save the spreadsheet as a CSV (comma seperated value) file and then use a text editor to manipulate it:
</td><td>).
<tr><td> to the first cell
and append a final </td></tr> to the final
cell on each row.
If you know Perl, you might also use a Perl script to generate simple
XHTML5 table entries from a CSV file. I have an example
Perl script (rename it
to csv2html.pl).
The column “All Available Video Clips” is not required (and should not be included) in the HTML5 table in the final report. It should still be contained in the spreadsheet you hand in.
You could, say, make a temporary copy of the spreadsheet where the column is deleted and save the CSV from there.
The column “Selected Video Clip(s)” is simply called “Video Clip(s)” in the HTML5 table in the final report.
Write a report of the heuristic evaluation using the skeleton report provided:
Some title information indicating the group number, the title/topic of the evaluation, and the name of each group member.
A summary of the evaluation and the most important findings written for your client's management. In real life, a manager would typically not have time to read the whole report, but would read only the executive summary to find out how the evaluation was done and what the main findings were.
The description of the evaluation scope and procedure should take up no more than 25% of the executive summary. Most of the executive summary should summarise the main findings.
A description of the methodology behind a Heuristic Evaluation, in your own words. Assumptions made about user characteristics and groupings and the extent of the evaluation.
A description of the evaluation environments (hardware, browser version, OS, date of evaluation, etc.) used by each evaluator.
Explain how each evaluator extracted or created video clips on each particular evaluation device or platform, and how they were edited and transcoded if necessary.
Discussion of three most positive findings, each illustrated with a video clip.
Aggregate table of positive findings in descending order of positivity.
Analysis and discussion of the five most severe problems (the top five from your sorted list), each illustrated with a video clip.
Aggregate table of problems in descending order of severity.
For the HE Report:
Use the skeleton report provided.
Follow any instructions and guidance embedded in the skeleton {the text in curly brackets}.
Remove any instructions and guidance {the text in curly brackets} from the file before you hand it in.
All text files (HTML, CSS, log files, etc.) must be encoded in UTF-8 format.
Make sure that when you edit such a file, when you save it, it is still in UTF-8 format. Check the settings on whatever plain text editor you are using.
Write straightforward, simple, valid XHTML5 using a plain
text editor
(single file, no frames, no Javascript,
no specialised HTML editor like Dreamweaver, no export from Word, no
conversion from LaTeX). Do not change the <section>
structure or the <section> ids. Do not make any
changes to the CSS file or add any <style> elements
to the HTML. Make sure your HTML is
valid XHTML5.
If you are not familiar with XHTML5, consult Chapters 10 and 11 of my INM 2014 lecture notes.
All video clips and images referenced in the report must be handed in as local copies (referenced by relative links), so that the report is self-contained. Do not link to such assets remotely.
Video clips for the top three positives and top five problems should
be given space in the report using the HTML5 video
element, as shown in the skeleton template.
By default, the HTML5 video element initially displays
the first frame of video. For our video clips, it is often more
appropriate to display a frame from somewhere within the video clip as
the initial image. This can be achieved by extracting a still frame,
for example n01-ka-links.jpg for a video clip
named n01-ka-links.mp4, and specifying it in
the poster attribute of the video element.
Remember, all the video clips which are included in the report must be no longer than 20 seconds duration, no larger than 10 MB, and no larger than FullHD resolution (1920×1080 pixels in landscape orientation or 1080×1920 pixels in portrait orientation).
Shorten or transcode any of the videos to be handed in with the HE Report which exceed these limitations.
Prepare a 15-minute slide presentation of your HE Report to show to your client at the client meeting (M2):
Make a separate presentation. Do not simply open your HE Report in a web browser and project that.
You may prepare your slide presentation in Powerpoint, Keynote, HTML, or PDF as long as you can export it (without loss of fidelity) to PDF.
Export your slide presentation to PDF and name
it he-slides.pdf
Open your PDF slide presentation with a PDF Viewer and make sure it can be used to give a presentation.
Make a directory called gT-GG-he for your heuristic
evaluation report, where T-GG is the number of your tutor and
group. For example, g1-05-he for Group 1-05.
Include your main file he.html and the CSS
file report.css.
Include your final, merged, and sorted spreadsheet
helist.xlsx.
Include the individual evaluation log files from each evaluator
in a subdirectory called logs.
Create a subdirectory presentation for your
presentation slides.
Place the video clips (and any corresponding poster images) selected
for your report into a subdirectory called videos.
All video clips referenced in your report must be handed in as local copies, so that the report is self-contained. Do not include video clips which are not referenced in the report.
When naming your files and directories, use only lower case letters, digits, and hyphens, from the 7-bit ASCII character set.
Do not use upper case letters, spaces, underscores, umlauts, or special characters. Only use dot (.) to designate a file's extension, nowhere else in a file name and nowhere in a folder name.
Your directory structure should look like this:
g1-05-he/
he.html
helist.xlsx
heuristics.pdf
report.css
logs/
log-hr.txt
log-ct.txt
...
presentation/
he-slides.pdf
videos/
p01-ka-pos01-breadcrumbs-work-well.mp4
...
n01-ka-neg01-links-yellow.mp4
...
Tidy up your directory. Do not leave junk files, backup files, etc. lying around.
Your directory must occupy 300 MB (300,000,000 bytes) or less. Check the size before uploading.
If your files are too big, you may have to delete something or reduce the size of something. Sometimes, you can significantly reduce the size of your slides PDF, without significantly affecting quality. Contact your tutor if you are unsure what to remove or make smaller.
Make a zip file of your directory and keep it elsewhere as a backup (do not upload it).
Submit your HE Report before the deadline to the Sapphire online submission system:
https://sapphire.isds.tugraz.at/
Ex 3 is a group exercise. The group makes a single submission as a group.
Every member of the group has access permissions to create and modify the group's submission, but it is best to designate a single person to do the uploading or any modifications, in order to avoid potential conflicts with other group members making changes in parallel.
Use the following approach to upload your files:
gT-GG-he/ (yes, the
folder, for example g1-05-he/, not all of the individual
files) and drop it into the upload area.
See my Guide to Using Sapphire.
Submissions will cease to be accepted 48 hours after the deadline.
At the client meeting, you must present the same version of your work which was handed in (uploaded to Sapphire).