August 20, 2004
Interactive TV user experience research
(another excerpt from my upcoming book: "Breaking Through to the Other side")
Some of the principles or methods from software and web usability work are applicable to i-TV -usability, but there are also differences, as the following sections will show.
Workshops and focus groups about concept sketches and prototype drafts
The pre-testing activities such as workshops and focus groups will give extremely valuable insight into the viewers’ preferences, motives, values and general behaviour. Viewers can tell you how they watch different types of programs e.g.:
• Are they gazing (watching the program with some interest) or are they glancing (only looking from time to time) or constantly going forth and back between these two positions, prompted by the content or the intensity of the program or the flow on the TV?
• Are they watching TV alone? When watching with someone, are they with the same viewing partner(s) or with different viewing partners?
Workshops and focus groups give viewers a chance to speak about what makes them turn to a specific channel or content, and what makes them turn it off. It is very important to capture these thoughts and preferences in order to build appropriate services and programming.
Of course, you will not know to what extent the test persons are conveying a complete image of their real behaviour – and they may not be lying but merely trying to adapt to the inherent values of the interview process such as portraying themselves as socially acceptable, coherent people. The closer to the real viewing environment the study is conducted, the better the image will be.
Still, the information that viewers will give you in a focus group or workshop will provide you with ideas and set the development off on the right track. The alternative to using focus groups and workshops is to develop the application or program based on assumptions and anecdotes that the developers and producers hold about what the viewers want and what the viewers are able to do. These assumptions may prove to be true (or not), but you will not know for sure until the end of a costly development process. Using focus groups and workshops is a proven way to get more for the money invested in the development of applications and programs.
Thinking aloud about a prototype or a running program
The thinking aloud testing method is applicable for usability testing of i-TV applications. Viewers can deliver meaningful comments as they watch or use the i-TV program. However, some adjustments must be made from the way that this method is used in software and web testing, as it is harder to follow the TV test persons selecting and pointing.
On the web or in software, users primarily use a mouse-button and the alpha numeric keyboard to give inputs to the system (thus called input devices). In i-TV , the viewers use a remote control that has more than twenty tiny buttons. In most web and software applications the cursor as well as the input from the user is shown on the monitor, making the user’s input directly accessible to observers. This is rarely the case in interactive TV applications.
Also, observers (including test leaders) can usually not see the test person’s remote control clearly. This makes it hard to see or infer which buttons on the remote control are being pressed. It is much harder to see which part of the input device is being used than in a web test. It may also make it harder for the test leader to determine the status of the i-TV application and, for instance, to determine exactly where the test person is in the application (which pages are displayed) and which options are available.
To get around these differences, the test person will have to speak continuously about where she is pointing and which buttons she is pressing. If the test person does not remember to do so the test leader will have to remind her. This can be stressful to both parties and the test person may not be really prepared to comment on her actions if for instance she is confused by something in the i-TV application. Also if she is using this medium for the first time she may not know what to expect and she may not yet have words for the features or the content she is experiencing or missing.
By Thomas.
August, 20. 2004.
Permanent URL to Interactive TV user experience research
August 09, 2004
Characteristics of usability and user experience
(excerpt from my upcoming book; "Breaking Through to the Other Side")
Usability is not the same as good design or trustworthy programming. Usability arises in the meeting between a user and the web site. The outcome of their meeting is fruitful if the user does not meet obstacles or barriers when performing her tasks.
If a web manager, the sender, was able to teach or personally guide all the users of that web site on how to use the web site to perform their tasks, success would be assured. Users, however, come to the web site pursuing their own goals on their own terms and without the benefit of a personal tutorial from the web manager.
The web manager faces the choice of either adjusting his web site to the users’ preferences, preconditions and requirements, or not getting the most out of his investment.
Characteristics of a usable system
A widely accepted and applied definition of usability comes from Jakob Nielsen’s book: “Usability Engineering” (published by Morgan Kaufmann, 1994). Originating before the World Wide Web became popular, this definition covers more than just the user interfaces found on the internet, and the definition consequently applies to IT systems in a broader sense.
According to that definition, a usable system should be:
• Easy to learn – the user should not need to pay much attention, use specific abilities or spend much time getting started;
• Easy to remember – the user should quickly and easily be able to access the system, whenever she wants to use it;
• Efficient – the user is able to perform relevant tasks;
• Reliable – the user can rely safely on the system and its solutions; and
• Satisfactory – the user likes the process and enjoys the experience.
The definition above describes usability as one of many qualities of a successful system, e.g. that the system is socially acceptable as well as accessible. The user must be able to deposit and receive information from the system whenever and wherever the user wants to and in a form acceptable to her.
The importance of each of the criteria varies from system to system – in a nuclear power plant control system Reliability means more than Satisfactory, and vice versa in a computer game.
According to the International Standard Organisation, ISO, usability is the "effectiveness, efficiency and satisfaction with which a specified set of users can achieve a specified set of tasks in a particular environment." (See also the paragraph on ISO 9241-11).
Characteristics of a user experience
A variety of elements contribute to the user’s experience including offline elements (e.g. media and marketing, shops and sales, support and delivery) and online elements like usability.
The focuses here is on the following criteria for the user experience on the web:
• The web site is easy to find on the internet;
• The web site is quick to enter;
• The web site meets the user’s requirements;
• The user can easily and quickly use the web site to solve her tasks in a satisfactory way; and
• The user wants to use the web site; and possibly re-use it again and again.
This definition focuses on the user’s ability and desire to perform her tasks by using a given web site. Can and will the user in fact perform her tasks on this web site? This is the question which decides the value and nature of the experience of the web site – to the user and consequently to the sender.
Today’s internet is rich in information and each individual web site will only be able to succeed if it grabs and maintains user attention. This aspect of engaging the user is not always included in the concept of usability. Often times the definition of usability is restricted to the users’ abilities to use the web site when in fact, a broader concept is needed. A qualitative assessment of the user experience should include a web sites’ ability to present and communicate information, create know-how, and give meaning to the user.
Several criteria are decisive for a user to choose to enter the internet and use a specific web site to perform her tasks. The concepts of usability and user experience both apply, making it hard to distinguish between the two. Here the general concept is usability but in many cases the term ‘user experience’ can be applied as well. Prominent aspects in both cases are:
• The user’s perception of being able to manage her communications;
• The content quality (including its relevance, actuality and trustworthiness) to the user; and
• The functional and visual appeal of the web site.
The content may be ever so relevant and interesting to a user, but if she cannot find it on the web site, then in practise the content does not exist in the world of that user. If users find it hard to operate the web site, or cannot manage the web site functions, they will blame the web site – even though the content or functionality that they seek exists.
By Thomas.
August, 9. 2004.
Permanent URL to Characteristics of usability and user experience
August 06, 2004
Usability definitions I love
"After all, usability really just means that making sure that something works well: that a person of average (or even below average) ability and experience can use the thing - whether it's a Web site, a fighter jet, or a revolving door - for its intended purpose without getting hopelessly frustrated."
- Steve Krug, Don't Make Me Think, 2000, p. 5
"Usability is de facto interdisciplinary, the bastard step-child of experimental psychology, information science, engineering, computer science, organizational psychology, business management, technical writing, and various other disciplines you might include. "
- Randolph G. Bias, Associate Professor, The University of Texas at Austin
By Thomas.
August, 6. 2004.
Permanent URL to Usability definitions I love
An old discussion still going strong: How many test persons do you need
Five to six test persons are commonly used for each round of usability testing. The effect of each test person’s work (namely the number and the seriousness of the problems revealed by the test persons) usually decreases considerably after the third or fourth test person. The test persons then often begin to repeat some of the problems that have already been uncovered by the previous test persons. The effect, of course, varies a lot from test person to test person and from website to web site. To web sites with a wide target group, each different segment can give very different results, and you will likely need three to four test persons from each segment in order to gather those differences.
To make sure that you collect enough data, it is usually a good idea to ask six test persons to the test, even though four are enough. The test persons may not show up for a variety of reasons. It is far preferable to carry out one test too many than risk having to cancel an otherwise well prepared test due to dropouts. As with focus groups and workshops, you can try to safeguard yourself against many dropouts by having substitutes ready.
An alternative to increasing the number of test persons is to use more than one test leader. Keep in mind that the test leader’s influences on test persons vary and lead to different results. Structuring the test by having one test leader test three test persons and another test leader test the remaining three can generate more insight than using the same test leader to test all six test persons.
I recently stumbled over this reader comment by Jim Lewis on HFI's website which seems relevant in this context.
"(...)Something that people always seem to forget (...) is that the use of small-sample usability tests is bound to iterative testing. If you're not iterating, then small samples don't make much sense. By the time you finish testing, you should have tested a relatively large number of participants -- not just five!"
Mr. Lewis also suggests a procedure to determine sample size (oh yes, the World is made of Math):
"Basically, the message was that the sample size you need depends on the problem discovery rate (p). If p is large, then you don't need very many people to discover the problems available for discovery. If p is small, then you need a larger sample. An ROI simulation indicated that the appropriate target for the proportion of problems to discover also depended on the value of p, with higher values having a break-even point at around 98% and lower values of p having a break-even point at around 86% (it's important to keep in mind that these values might depend on the assumptions made for the simulation -- but I think they are still informative). The data I reported in Lewis (1994) had p=.16, so to get to 86% problem discovery, I needed to run 12 participants (I actually ran 15)."
Interesting, but I really don't know what p is - is it any close to pi (hahaha)?
Sounds a bit like: if it is hard to find the problems you need more testing. "One size doesn't fit all but try twelve anyway".
My law? Start with small sample (three), test, evaluate, continue testing if the evaluation results were poor, iterate. Good luck.
By Thomas.
August, 6. 2004.
Permanent URL to An old discussion still going strong: How many test persons do you need
August 04, 2004
Copenhagen as a tourist destination - Sixteen top three lists
Neighbourhoods are in () – neighbourhood map:
1. Getting around Copenhagen
Walk. It is a small town (a least the parts you wish to see as a tourist are within 20 minutes walk from Amagertorv). See Copenhagen Walking Tours for inspiration.
Get yourself a bike. Deposit a DKK 20 coin (2,5 euro), ride away and get the coin back after the trip. ("Bycyklen")
City maps by krak.dk in English.
Public transport travel planner rejseplanen.dk also in English.
Canal tours in the harbour.
2. The fairy tale city
Tivoli after dark (mid April -mid September and November 12 – December 23) (city, by the central station).
Kongens Have (city, by Rosenborg Castle).
3. The metropolis
Taking the new metro is an experience in itself (especially when it works).
The city museum and the museum street (Vesterbro).
4. The Royal city
Rosenborg Castle – the Royal Danish Collections (City).
Amalienborg Palace – the winter residence of Queen Margrethe and Prince Henri (City).
5. Art collections
The Glyptoteket – Carlsberg’s own collection (City, next to Tivoli).
Thorvaldsens Museum – Golden age sculptor (City, next to the parliament).
Hirschsprungs Collection – Golden age collection (Østerbro, next to the National Gallery of Art).
6. Art exhibitions
Louisiana Museum of Modern Art – okay, it is not Copenhagen, but it’s so nice (and close – 20 minutes North by train).
Charlottenborg Exhibition Hall (City, Kongens Nytorv).
7. For kids aged 6 -16 years
Experimentarium (in Hellerup, on the Tuborg Havn harbour).
Skateboards etc (Østerbro, in the Fælled park). (I am looking for a link to this).
Tivoli (City).
In winter:
Skating rinks on Kongens Nytorv, in Tivoli and other places.
Østerbro has an indoor play court in Remisen (link to follow).
8. Copenhagen specialities (no admissions)
Christiania – the free city (Christianshavn).
A walk on the ramparts of Kastellet (City, near the little mermaid) or around Christianshavn (start at Lille Mølle).
9. ***** Dining
Søren K in the Royal Library (City).
Era Ora (Christianshavn – ten minutes walk east of Kongens Nytorv).
10. Dining Very Danish
M.G. Petersens gamle familiehave (since 1858, Frederiksberg, near the zoo).
Cap Horn (City, in Nyhavn).
Bastionen og Løven (Christianshavn).
11. Dining in my neighbourhood, Østerbro
Circus, Rosenvængets Alle.
Saint Jacques, Sankt Jakobs Plads.
Nørrebro Bryghus, Ryesgade (near Sankt Hans Torv).
12. Accommodation
For a homey feel to a longer stay, try Ni små hjem (Østerbro)
13. Tourist traps (be warned)
Strøget (city – the pedestrian street)
14. Copenhagen tourist resources in English
The Copenhagen Post - news and an excellent In&Out Guide.
Aok.dk - an online cityguide.
Frommer's has a large portion on Copenhagen.
The airport.
We do talk a lot about the weather in this country. For good reasons. This site has daily photos and temperatures from Copenhagen.
Wonderful Copenhagen is the official tourist portal.
Krak Tourist
15. Propaganda
Living in Copenhagen
16. This is Denmark
Map - just fill in the bridges between Fyn (Nyborg) and Sjælland (Korsør) and between Denmark (Kastrup) and Sweden (Malmø).
More maps.
The Ministry of Foreign Affairs sponsors denmark.dk
Clickable map.
The municipality offers this online address search in Danish and the entire most recent Municipal Plan 2001 in English.
By Thomas.
August, 4. 2004.
Permanent URL to Copenhagen as a tourist destination - Sixteen top three lists