Cognitive Daily

Last week’s Casual Fridays study was inspired by my annoyance at a website form which required me to constantly switch between typing in information and selecting it from a menu. I wondered if there was really any significant benefit to requiring the use of menus, when typing (for me, anyways) seemed so much faster.

So we developed two versions of the same simple 8-question quiz, one of which required users to alternate between menu-responses and typed responses, and the other which allowed respondents to type in each response. We asked respondents to answer the questions as quickly as possible. About 1,200 people participated. The point of the quiz was to have obvious questions with easy answers, to simulate entering personal information on a form without requiring you to reveal a bunch of actual personal information. For the most part, we succeeded, with about 97 percent accuracy on the quiz responses.

But did the type of quiz affect how quickly respondents answered? It did, a bit:

The respondents using menus took an average 75 seconds to finish the quiz, while the fully-typed responses averaged only 66 seconds. That’s about a 15 percent speed difference, or nine seconds per respondent. Put another way, we wasted about an hour’s worth of our respondents’ time by requiring them to use menus.

On the other hand, I had to spend some time cleaning up the data: One question asked “What country features the Grand Canyon, the White House, Disneyland, and Florida?” Typed responses included “U.S.,” “U.S.A.,” “The United States,” “United States of America,” and “usofa.” I had to convert all those responses to “United States” to match the menu responses. In fact, I spent about an hour fixing up the data, so depending on whose time you value more, the time savings of typing may not have paid off at all.

But even after I sorted through the data to make the typed responses consistent with the menu responses, there was still a difference in error rate:

On this graph, the green bars represent the questions that could be answered with menus (for the “menu” group). The purple bars show the error rate on questions that both groups had to type. Respondents using menus made more than four times fewer errors than those who typed their responses. The menu group was even a little more accurate than the typing group on the questions that everyone had to type.

Overall, the menu group made less than half as many errors as the typing group.

What’s especially interesting to me is the type of errors that were made by the typing group. Of course there were typos, but there were also blunders based on inattention. For the “What country features the Grand Canyon, the White House, Disneyland, and Florida?” question, answers included California, D.C., and Arizona. This clearly results from a misreading of the question as asking about a state rather than a country. For another question we asked “What country features the Eiffel Tower, Paris, and the Louvre?” Quite a few respondents answered “Paris,” again clearly misreading the question. Both of these types of responses simply wouldn’t have been allowed in the menu group, which provided only a list of countries.

Menus also cut down on the number of jokesters responding. For example, one respondent who claimed to be 120 years old finished the survey in just 38 seconds! It might be amusing to me as I compile the survey responses, but I imagine a clerk processing an online purchase would be less amused by pranks like that. By reducing the number of possible responses, menus help tame data sets and generally make life easier for those processing the data.

One aspect of our results is quite relevant for psychology researchers: we see clear evidence here of a speed-accuracy trade-off. The faster respondents were significantly less accurate than slower ones. In many studies measuring reaction times, it’s important to see if there’s a trade-off between speed an accuracy. Often if such a trade-off exists, it nullifies the study results. In this case, that’s clearly part of what’s going on: the slower menu-based respondents were more accurate not only on the menu-based questions, but on typed responses as well. But I don’t think it nullifies the results. Yes, menus do slow you down a little bit, but the reward is a dramatic improvement in accuracy.

One more thing: The study was based on my annoyance with forms that mix menu items and typing, but perhaps not all people share my disdain for menus on forms. Let’s make this a poll:

Comments

#1 Rosie Redfield
September 18, 2009

I prefer menus because then I don’t waste MY time figuring out what form the answer should take.
#2 name
September 18, 2009

Keyboard shortcuts help a lot. I just used tabs, arrows, and typed “Unit…” until it appeared in the long menu.
#3 NoAstronomer
September 18, 2009

One question that always drives me nuts on forms is having to use a dropdown menu to enter year (or age). For one application I was involved in developing recently the user insisted on a dropdown for year as part of date-of-birth. Right up until she came to test the application and realized that a dropdown with 110 entries was a *bad* thing.

Her year wasn’t even as far down the list as mine was.

Mike.
#4 Paul
September 18, 2009

Clearly you are not a programmer.

Logic can be accurately applied to menu responses without the code taking all possible misspellings and synonyms into consideration.
#5 Steve
September 18, 2009

It’s not typing or menus that make me more or less annoyed with a form, it’s other formatting elements, length, and contents.

- The tab key should move you from item to item in a logical and obvious order.

- Whenever possible, all menu items should be present without reloading. Making me wait for a reload between questions is annoying.

- Only ask for information that you have a plausible reason to use. (My bank needs daytime and evening phone numbers. My favorite blog does not.)

- If I “submit” with an error or blank field, don’t delete my other answers from the form.
#6 speedwell
September 18, 2009

I support an engineering database with thousands of users. Several years ago, when the database software we use was a new implementation, we allowed the users to specify the attributes of products by typing in the values (for example, a “size” attribute might be expressed as “X”, “X inches”, “X meters”, etc.). Data cleanup and cross-referencing was a nightmare, with well over a half million products in the system.

Now we have a million products in the system. We also have a team dedicated to administering the sets of attributes required for each product (a pen, for example, might have the attributes “material,” “ink color,” and “vendor”) and the list of allowed values for each attribute (for ink color, the list might include “red,” “black,” “blue,” and “green”). The attributes are set for each type of product. The allowed values are usually selected from a drop-down list. There are standards for the few remaining free-entry fields, and we force a check step before the information can be used (so the products can be proofread by another user and standards can be enforced).

Bottom line: for twice as many products, we have added only two more database admins, and more time per admin to address higher-level tasks.

The engineers complained that we were taking away their right to specify what they wanted. Durn tootin’ we were. You can’t search for a piece of pipe in the system if you’re searching by size, and the sizes for a quarter-inch pipe are entered as “0.25,” “0.250″, “1/4″, and “250″, combined with “in”, “inch,” “[inch symbol]“, and “inches”.
#7 JohnV
September 18, 2009

We went through this with some scripts being written to facilitate the entry of experimental descriptions into a database linking it with automatically generated data.

The first script our interns whipped up was text entry. I put the breaks on that by asking them to spell the names of the 3 bacteria that would be used for these experiments, then showing them the correct spelling :p
#8 speedwell
September 18, 2009

JohnV: W00t! You are my database admin Hero of the Day.

Steve: We have an attribute editing dialog that actually keeps the original values off to the side so if you change them, you can see what you changed them FROM. The new values don’t get applied until you click the OK button. So we don’t have any more errors of the “I changed the wrong thing and I forgot what to correct” variety.
#9 Marius
September 18, 2009

Safari lets you type out drop-down menus anyway, I don’t see it as much of a problem. I’m pretty sure most modern browsers yields the same behaviour.

If I tab my way to a birth date menu, I just type 1986 for year and tab to the next field. In fact, this ensures data integrity and is faster than plain text fields, since february is the only month starting with F.
#10 Dave Munger
September 18, 2009

Clearly you are not a programmer.

Logic can be accurately applied to menu responses without the code taking all possible misspellings and synonyms into consideration.

Indeed I am not a programmer. However, one hour is the amount of time it took me to use logic and search/replace functions to find all the misspellings/synonyms. It would have taken much longer if I had done it by hand — 1,200 participants X 8 responses each!
#11 Jennifer Leigh
September 18, 2009

My preference is for menus that I can tab into, and have them jump to the entry I’m typing out. I hate mousing, but I understand the need for accuracy.
#12 Dave Munger
September 18, 2009

For those suggesting keyboard shortcuts for menus, they may help, but we have empirical evidence that menus are slower.

This might be due to the fact that not everyone knows the shortcuts. Not everyone is a touch-typist either.

Sometime in the future maybe we’ll test folks using men shortcuts against straight typists. I’d still submit that a fast typist is faster than a fast menu-user.
#13 becca
September 18, 2009

@Marius — Safari does have the behavior; however, IE in particular does not and some forms require IE — IE if you type “UN…” it will select the first item under U and then the first item under N so typing “UNITED STATES” will result in whatever country starts with S in the menu.
#14 Jessica
September 18, 2009

Technically, the y axis of your plot should be “time” not “speed”. If menus take longer, the bar for speed would be lower, not higher.
#15 Tony P
September 18, 2009

Good forms let you start typing in a menu pulldown to narrow the selection. Those are my favorites.

Tend to not take my fingers off the keyboard and those instances.
#16 Richard
September 19, 2009

With Firefox, it’s not just the first letter of menu items you can type when you tab to a dropdown – you can just type the whole thing, and if it’s there, you can find it. So to enter my birth year I just type it in, and Firefox finds it for me and selects it.

What it means is that I can tab-and-type my way through a form, with the added protection of only being able to enter valid data.

I’m not sure if Internet Explorer does this though – I think it might treat every keypress as a potential match for the first letter of a menu item.
#17 JanetK
September 19, 2009

Maybe you should have talked to your nearest database designer or form designer or any who is involved in collecting and storing computerized data. Good heavens – what a predictable result!
#18 Zachary Constantine
September 19, 2009

Usability has come a long way from the time whomever threw together the survey site coded their forms – the good news is that there are viable alternatives (none of which appeared to have been implemented on the test site).

First and foremost, text-entry is great when accompanied by an AJAX drop-down menu (try a search at Google to see what I mean) – too many users are oblivious to the fact that they can begin typing their desired item in the drop-down menu and the AJAX drop-down forces this behavior gracefully.

Secondly, asynchronous form validation should be occurring on any reasonably-usable form.

These considerations alone would be sufficient to eliminate errors and semantic cleanup overhead and the Javascript-disabled pathway (though slightly less elegant) follows from a little design savvy.
#19 Jordi
September 20, 2009

At first I was really surprised at the speed results, but then I realized that some people are actually using the mouse to scroll through the drop-down lists and select their item. This is clearly slower.

Both drop-down lists and text boxes should be combined with typing and if we want the user’s response to be valid, there is no way in hell that the text box would be faster. Why? Because you simply have to type less in the drop-down list, but if you want, you can type just as much. The only way text boxes would be faster, is if you want to allow answers like US, U.S., USA, etc. But this would mean your job interpreting these answers is that much harder that I submit it is almost never worth it.
#20 Jordi
September 20, 2009

Sorry for posting twice, but I forgot to say that it might be interesting to see some boxplots of the speed performance, because I have a very strong feeling that the fastest menu users should be faster than the fastest typists.

Also, it might be intesting to see some correlations between speed and accuracy, because I imagine that the menu users’ higher accuracy might also be due to the fact that they have the option to see the possible answers (and not just the fact that they were slower on average).
#21 Michael Dickens
September 21, 2009

Maybe the best thing would be a limited and “smart” typing box. It only lets you type certain answers, and if you type the first letter of a word it guesses what you want to type. For example, if you type “u”, it guesses “United States”. And if you have to put in your date of birth, you can only enter a two-digit number. This requires the user to be a little smarter, but I think we can handle it.
#22 R Simmon
September 21, 2009

Michael Dickens nailed it: the correct answer (for constrained datasets) is autocomplete. One of my favorite interface design books: Designing the Obvious: A Common Sense Approach to Web Application Design by Robert Hoekman Jr. covers the specifics well.
#23 Joe DiPilato
September 21, 2009

there are ~(8) different group types that took this quiz.
they are a combination of:
fast or slow users
efficient or inefficient users
menu or text field users

(where efficient users are those who navigate through forms via shortcuts. efficient browsers unlike IE play a large roll in menu usage speed)

(fast users refers to users who are able to come up with the answers and type quickly)

I believe that the speed at which the different groups could fill out the forms would be as follows, where higher numbers represent faster completion times:
1)fast efficient menu users
2)fast efficient text field users
3)slow efficient menu users
4)fast inefficient text field users
5)slow efficient text field users
6)slow inefficient text field users
7)fast inefficient menu users
8)slow inefficient menu users

the majority or users fall into the inefficient group,
therefore 4)fast inefficient text field users would be the highest speed status that they could attain and menu users would almost always lag behind.

efficient users slowest times would be 5)slow efficient text field users and fastest times being 1)fast efficient menu users.

I believe true efficiency plays a much larger role than speed or field type.
(unfortunately true efficiency can not be tested on users who have just learned efficient methods because the time it would take to remember how to use them would nullify the time saved through using the efficient methods)
#24 Suzanne Hillman
September 21, 2009

When menus in forms are correctly set up such that you can type the word and cause it to go to the word in question, I am most happy, as this improves speed and accuracy. (clicking is evil for my wrists, which is why I try to type for menus)
#25 Steph
September 21, 2009

I am curious as to the statistical significance of the difference between the groups. Do you calculate this? I think that posting p-values would be very useful.
#26 Michelle
September 21, 2009

This expt, which falls into the field of Usability or User Experience Research, was trying to find out how usable menus vs text fields are, although I believe it really only applies to the specific form that was presented. This is because, in theory, menus and text fields are both successful when used in the appropriate context, and with the supporting usability improvements.

As mentioned above, web forms vary a lot, and the usability of forms can be greatly improved. For instance, there are text fields with auto-suggest or auto-completion; inline form validation which prevents users from submitting a form with errors or invalid data; logically populating the menus and having some values pre-selected (such as date of birth not listing 1900-2000 and having an average date selected), and others.
#27 David
September 28, 2009

Typed responses included “U.S.,” “U.S.A.,” “The United States,” “United States of America,” and “usofa.” I had to convert all those responses to “United States” to match the menu responses.

I assume your study doesn’t really matter and nobody really cares if you get it right.

In the pharmaceutical industry, modifying the originally entered data would be unacceptable. Besides leading to “not approved” from the FDA, there would be NEJM articles about how pharma doctors the data to make it fit.
#28 Chris H.
October 2, 2009

A very interesting study–it is good to see some data to back up the assumptions.

Two cases where I think you could forego the select menu, one of which has been mentioned already:

- Year of birth. As someone mentioned, a list of years since 1900 is ridiculous. Yes, someone could foolishly enter just the last 2 digits even though you make the field 4 characters wide and specify “YYYY”. Those users get a friendly error message asking them to correct it.

- State abbreviation for a mailing address. Everyone should know the proper abbreviation, and even if they don’t–you should be able to validate against the ZIP code. Sure, savvy users know they can tab from the City field to the State field and type “mmmm” to get MI (Michigan). But for the majority of users, what a pain to scroll through the list to enter 2 characters!

In these two cases, the data integrity can be preserved without slowing/inconveniencing the user. There are probably many others I am overlooking.

Some would argue that we should remove the State field, since it is superfluous with the ZIP code field–but that interrupts the address-entry flow many users have by now. One case where including the State menu makes sense is if the form is not necessarily asking for the user’s address–entering a shipping address, for example: you might remember that Aunt Martha lives in Minnesota, but might not remember that MN is the abbreviation.