First of all, read the sections in the lecture notes on heuristic evaluation. You will have to explain what these techniques are to your client (tutor) at the first client meeting.
This exercise only involves writing the plan, you do not do the actual evaluation yet!
Preparation:
The four (or three) members of your group will be the evaluators for the heuristic evaluation.
The web site which has been assigned to your group will be visible in TUGRAZonline in the comment field next to your group number.
Unless otherwise notified, assume that you will be evaluating the entire web site.
Generally speaking, do not follow links to external web sites and
evaluate them too. However, it often makes sense to follow links to
subsites and consider them too (say, tickets.site.com).
Check with me or your tutor if you are unsure.
Most of the chosen web sites are available in German. Some of the chosen web sites are available in English and German. Sometimes, some of the web sites are available in English only.
If the web site has both English and German versions, each evaluator should pick one language (English or German) for the bulk of their evaluation, but then also take a look with the other language too.
Consider the target user population(s) for your assigned web site. Are there distinct user groups with distinct needs and goals?
If the web site has several distinct user groups, you should consider all of the potential user groups in the heuristic evaluation.
Discuss the goals the users from these groups might have and typical tasks they might want to perform on the web site.
Adapt the HE Plan materials as necessary for your evaluation (see below).
Plan for all of your evaluators to work at around the same time, to minimise the risk that the web site changes in between evaluations.
Write a short plan for the heuristic evaluation:
Some title information indicating the group number, the title/topic of the evaluation, and the name of each group member.
A description of the methodology behind a Heuristic Evaluation, in your own words.
User profiles describing which groups of users you think are supposed to be addressed by the web site. Describe the goals and typical tasks for each user group.
The names, characteristics, and environments of each evaluator.
Coordinate between yourselves, so that the evaluators use a mix of devices and browsers to inspect the web site:
Two of the evaluators should evaluate on a mobile device (= smartphone or tablet, running Android or iOS), in portrait orientation, each using a different approved mobile web browser.
On mobile, the approved browsers are: Chrome, Firefox, Safari, and Samsung Internet Browser.
The other evaluator(s) should evaluate on a PC (= desktop or laptop, running Windows, MacOS, or Unix), in landscape orientation, each using a different approved PC web browser.
On PC, the approved browsers are: Chrome, Firefox, and Safari.
It is OK for one evaluator to use (say) Chrome on PC, and another evaluator to use Chrome on mobile, since they are different devices.
You must use real devices and not an emulator or simulator.
If you are using a mobile browser and your web site suggests you should go to the mobile version of the site, then do so. However, do not install a native app, even if your web site suggests you should do so.
In an attempt to reflect real usage patterns, one of the evaluators on mobile and one of the evaluators on PC should plan to use an ad blocker.
If you find a problem which may be due to blocked content, unblock ads temporarily to see whether the problem persists. If the problem then goes away, document it as a problem apparently caused by ad blocking.
Before you start evaluating, plan to reset the browser to a fresh state:
Make sure all add-ons and extensions are disabled (except for an ad blocker, if you are using one via an extension)).
Delete all cookies.
Delete all temporary files (clear the cache).
Plan to try out your assigned website first with all cookies enabled and then with most cookies disabled (only necessary cookies enabled) to see if there is any difference.
If you find a problem which appears to be due to disabling cookies, document it as such.
Explain how you will make make screen video recordings on the various platforms you have chosen to evaluate. See my Guide To Session Capture on various platforms. You should plan to record your voice (with a live commentary while you work) as well as what happens on screen.
On touch devices, see if your recording software can turn on display of touch events.
The screen recording should be at most at FullHD resolution (1920×1080 pixels in landscape orientation or 1080×1920 pixels in portrait orientation). If your device has higher resolution than FullHD, please plan to adjust the settings or make the browser window smaller. If your device has lower resolution than FullHD, record at its maximum resolution. The standard browser GUI should be included in the recording.
If the native resolution of your device is too high in one or both directions, and you cannot adjust the settings or make the browser window smaller, you will have to record at higher pixel resolution and later transcode down uniformly to at most FullHD. Do not distort the video and do not create black or empty strips to either side (or top and bottom). See my Guide to Video Transcoding. Unfortunately, transcoding always results in a loss of quality.
Use a good microphone, and make a test recording to make sure everything is working and the audio can be heard.
Our preferred video format is MP4 with H.264 video and AAC audio. If your device/software can record in that format natively, that will be perfect. Otherwise, you will have to look at converting/transcoding your video clips after extraction.
Plan to use the following recommended video settings:
| Container: | MP4 |
| Output Video Resolution: | 1920×1080 (FullHD) [or 1080×1920] or lower |
| Frame Rate: | 20 |
| Codec: | H.264 |
| Rate Control: | VBR (Variable Bit Rate) |
| Bit Rate: | 5000 Kbps (= 5 Mbps) |
Plan to use the following recommended audio settings:
| Channels: | Stereo |
| Codec: | AAC (= mp4a) |
| Sample Rate: | 44100 Hz |
| Bit Rate: | 160 Kbps |
| Bits per Sample: | 32 |
Explain how you will extract video clips from the screen capture videos. All findings must be illustrated with a video clip.
In Ex 2a, each evaluator will only be handing in their recreated/extracted video clips, not their entire screen capture videos.
Adapt and include the log files you will be using for the evaluation:
Assign each evaluator a two-letter shorthand code based on their initials. For example, Keith Andrews would be “ka” in lower case and “KA” in upper case.
If two evaluators in your team both have the same initials, then assign one of them a variant. For example, Ken Anderson might be assigned "kn" in lower case and “KN” in upper case.
Copy the skeleton log file
log-ee.txt
from the materials to create a plain text log file
for each evaluator:
log-EE.txt
where EE is replaced by the initials of the evaluator in
lower case. Do not use any special characters, umlauts, or spaces in
the file names.
For example, if your name is Keith Andrews, the log files should be
named log-ka.txt.
Each evaluator should fill in the metadata at the start of their log file to match their (planned) evaluation environments. Enter the name of the corresponding evaluator. Take care to preserve the character encoding of the log files as UTF-8.
Each negative finding (problem) will be assigned an ID of the
form EE-Neg01, EE-Neg02, etc. Each positive
finding will be assigned an ID of the
form EE-Pos01, EE-Pos02, etc. In each log
file, replace the initials “EE” in the example problem and positive
finding IDs with the initials of the corresponding evaluator in upper
case.
Adapt the example spreadsheet
helist.xlsx
for your evaluation:
In the sheet (tab) “Problems”, replace the generic codes E1 through E4 with the upper case initials of your evaluators (yourselves) in the columns “Found By” and “Severity”.
In the sheet (tab) “Positives”, replace the generic codes E1 through E4 with the upper case initials of your evaluators (yourselves) in the columns “Found By” and “Positivity”.
For the HE Plan:
Use the skeleton provided.
Follow any instructions and guidance embedded in the skeleton {the text in curly brackets}.
Remove any instructions and guidance {the text in curly brackets} from the file before you hand it in.
All text files (HTML, CSS, log files, etc.) must be encoded in UTF-8 format.
Make sure that when you edit such a file, when you save it, it is still in UTF-8 format. Check the settings in whatever plain text editor you are using.
Write straightforward, simple, valid XHTML5 using a plain
text editor
(single file, no frames, no Javascript,
no specialised HTML editor like Dreamweaver, no export from Word, no
conversion from LaTeX). Do not change the <section>
structure or the <section> ids. Do not make any
changes to the CSS file or add any <style> elements
to the HTML. Make sure your HTML is
valid XHTML5.
If you are not familiar with XHTML5, consult Chapters 10 and 11 of my INM 2014 lecture notes.
Prepare a 5-minute slide presentation of your HE Plan to show to your client at the client meeting (M1):
Make a separate presentation. Do not simply open your HE Plan in a web browser and project that.
You may prepare your slide presentation in Powerpoint, Keynote, HTML, or PDF as long as you can export it (without loss of fidelity) to PDF.
Export your slide presentation to PDF and name
it heplan-slides.pdf
Open your PDF slide presentation with a PDF Viewer and make sure it can be used to give a presentation.
Make a directory gT-GG-heplan for your heuristic
evaluation plan, where T-GG is the number of your tutor and
group. For example, g1-05-heplan for Group 1-05.
Copy the skeleton plan and other files into this directory.
Your main file should be named heplan.html.
Your spreadsheet should be named helist.xlsx.
Create a subdirectory logs for your skeleton log files.
For each evaluator, there should be one log file
named log-EE.txt, where EE are the initials
of each evaluator in lower case.
Create a subdirectory presentation for your
presentation slides.
When naming your files and directories, use only lower case letters, digits, and hyphens, from the 7-bit ASCII character set.
Do not use upper case letters, spaces, underscores, umlauts, or special characters. Only use dot (.) to designate a file's extension, nowhere else in a file name and nowhere in a folder name.
Your directory structure should look something like this:
g1-05-heplan/
helist.xlsx
heplan.html
heuristics.pdf
report.css
...
logs/
log-hr.txt
log-ct.txt
log-tg.txt
log-ss.txt
presentation/
heplan-slides.pdf
Tidy up your directory. Do not leave junk files, backup files, etc. lying around.
Your directory must occupy 20 MB (20,000,000 bytes) or less. Check the size before uploading.
If your files are too big, you may have to delete something or reduce the size of something. Sometimes, you can significantly reduce the size of your slides PDF, without significantly affecting quality. Contact your tutor if you are unsure what to remove or make smaller.
Make a zip file of your directory and keep it elsewhere as a backup (do not upload it).
Submit your HE Plan before the deadline to the Sapphire online submission system:
https://sapphire.isds.tugraz.at/
Ex 1 is a group exercise. The group makes a single submission as a group.
Every member of the group has access permissions to create and modify the group's submission, but it is best to designate a single person to do the uploading or any modifications, in order to avoid potential conflicts with other group members making changes in parallel.
Use the following approach to upload your files:
gT-GG-heplan/ (yes, the
folder, for example g1-05-heplan/, not all of the
individual files) and drop it into the upload area.
See my Guide to Using Sapphire.
Submissions will cease to be accepted 48 hours after the deadline.
At the client meeting, you must present the same version of your work which was handed in (uploaded to Sapphire).