Odd & Annoying Survey Methodology

Our municipality has contracted a survey firm to evaluate the after-school activities for children that it supports. Circus school, piano lessons etc. Questionnaires have been sent to (some? all?) enrolled children.

My kid is in three of these activities. I got three almost identical questionnaires, interpreted them as a mail-merge glitch, responded to one and threw two away unexamined. Then I was nagged about those two.

If I were running the survey I would purposely avoid collecting data on the same kid for more than one activity.

Tags

More like this

I grew up in the days of the SNES and the Sega Megadrive. Even then, furious debates would rage about the harm (or lack thereof) that video games would inflict on growing children. A few decades later, little has changed. The debate still rages, fuelled more by the wisdom of repugnance than by data…
When I was in elementary school, we had two recesses every day: 20 minutes in the morning and 20 minutes in the afternoon. Plus, we had a 30-minute lunch period, and as soon as we finished eating we were free to go outside and play until the bell rang. So I was a little surprised when our kids…
It's long been recognized that American kids suck at math, at least when compared to kids in Singapore, Finland, etc. What's less well known is that the steep decline in proficiency only starts when kids are taught algebra. That, at least, is the conclusion of a new government report: "The sharp…
Marie-Claire Shanahan is an Assistant Professor of Science Education at the University of Alberta, in Edmonton, Alberta, Canada. As a former science teacher, she was always surprised by the ways that students talked themselves out of liking science - and she decided to do something about it. She…

-Are you still in Saltsjöbaden? If the municipality is small enough, they might actually pay attention if you give them advice about survey methods. In larger places, your complaints will just drown in white noise.

By Birger Johansson (not verified) on 30 Nov 2012 #permalink

Saltsjöbaden was a separate municipality between 1909 and 1970. Before and after that interval it has been part of Nacka municipality, pop. 93,000.

Does the survey firm have any way of knowing that John Smith the piano student and John Smith the football player are the same person?

I am in a small municipality (population 1X,XXX), and my name is not common in this part of the US. However, there was a period of several years when somebody sharing my first and last name was also on the local voter rolls. Every ten years the state requires municipalities to purge their voter rolls of non-voters (people who move away, especially out of state, do not always inform the town clerk in their former municipality). I had been voting (and still do vote) in every election, but my namesake had not. The town clerk sent me the note saying that I had to confirm my registration in order to stay on the voter rolls.

I don't know how common name duplication is in Sweden, but if it is in any way similar to English I expect there is a non-negligible chance in a municipality the size of Nacka that there are two people with the same name.

By Eric Lund (not verified) on 30 Nov 2012 #permalink

This can be based on the fact that different databases do not work together and cannot see each child as uniquie, in database model terms. Nor can the system see the relation betwwen the child and the parents. We have our system of unique person identification numbers here in Sweden but sometimes you are not allowed to use that across different database systems except for taxing or crime investigations.

It is based on different databases not being able to talk with each other or badly designed ones.

By Thomas Ivarsson (not verified) on 30 Nov 2012 #permalink

If they care to do that check, Obviously the did not! Information is not connected or consolidated in their systems.

Dead people get offers sent to them!

By Thomas Ivarsson (not verified) on 30 Nov 2012 #permalink

Are they allowed to use citizen numbers that Martin? In Canada, for very good reason it is illegal to use people's government-issued ID numbers as a general-purpose identifier. Allowing that, as they do south of the border, would make it too easy for companies to surveil people and combine databases without people's consent; it is also dangerous because people tend to assume that such a number must be strong ID, and if they give it out widely it is easy to steal. So while a Canadian city might have such data for taxpayers (I am not sure), it would not have it for everyone and would not be able to give that data to a private marketing company.

You can blame the research firm (after all, they are the ones responsible for administering the survey and advising the customer on how to do it right); however, having had some experience with market research, I would guess:

a) The survey company was given three separate lists for the surveys and wasn't paid extra for de-dupping the lists
b) The three surveys may be tabulated separately
c) The customer may have insisted on doing it this way

And unfortunately I am not a survey methodologist nor am I a statistician by education or training I cannot comment on whether or not it is actually valid to survey you once for each activity. It probably depends on the questions asked; though, one would think it would be better to ask them all in one survey rather than three nigh duplicate surveys.

However, think about a survey asking about soft drinks. They are going to ask you a series of questions about coca cola, then pepsi, then RC Cola, then whatever else is next. The questions are going to be identical but the product is going to be different. This is probably analogous to sending you one survey for each activity.

By Felix Gutt (not verified) on 02 Dec 2012 #permalink

Trouble is, polling a person changes that person. My replies to a questionnaire will be different if it's the third identical one that day than if there's only one.