Odd & Annoying Survey Methodology

Our municipality has contracted a survey firm to evaluate the after-school activities for children that it supports. Circus school, piano lessons etc. Questionnaires have been sent to (some? all?) enrolled children.

My kid is in three of these activities. I got three almost identical questionnaires, interpreted them as a mail-merge glitch, responded to one and threw two away unexamined. Then I was nagged about those two.

If I were running the survey I would purposely avoid collecting data on the same kid for more than one activity.

Comments

  1. #1 Birger Johansson
    November 30, 2012

    -Are you still in Saltsjöbaden? If the municipality is small enough, they might actually pay attention if you give them advice about survey methods. In larger places, your complaints will just drown in white noise.

  2. #2 Martin R
    November 30, 2012

    Saltsjöbaden was a separate municipality between 1909 and 1970. Before and after that interval it has been part of Nacka municipality, pop. 93,000.

  3. #3 Eric Lund
    November 30, 2012

    Does the survey firm have any way of knowing that John Smith the piano student and John Smith the football player are the same person?

    I am in a small municipality (population 1X,XXX), and my name is not common in this part of the US. However, there was a period of several years when somebody sharing my first and last name was also on the local voter rolls. Every ten years the state requires municipalities to purge their voter rolls of non-voters (people who move away, especially out of state, do not always inform the town clerk in their former municipality). I had been voting (and still do vote) in every election, but my namesake had not. The town clerk sent me the note saying that I had to confirm my registration in order to stay on the voter rolls.

    I don’t know how common name duplication is in Sweden, but if it is in any way similar to English I expect there is a non-negligible chance in a municipality the size of Nacka that there are two people with the same name.

  4. #4 Martin R
    November 30, 2012

    I’m pretty sure they do this by citizen number.

  5. #5 Thomas Ivarsson
    Malmö
    November 30, 2012

    This can be based on the fact that different databases do not work together and cannot see each child as uniquie, in database model terms. Nor can the system see the relation betwwen the child and the parents. We have our system of unique person identification numbers here in Sweden but sometimes you are not allowed to use that across different database systems except for taxing or crime investigations.

    It is based on different databases not being able to talk with each other or badly designed ones.

  6. #6 Martin R
    November 30, 2012

    You could nevertheless compare name & address and see that it’s the same person.

  7. #7 Thomas Ivarsson
    November 30, 2012

    If they care to do that check, Obviously the did not! Information is not connected or consolidated in their systems.

    Dead people get offers sent to them!

  8. #8 Birger Johansson
    December 1, 2012

    (OT) ”Lair of Unicorn Found In North Korea” (Swedish-text article) North Korean propaganda; the gift that keeps on giving http://www.dn.se/blogg/markligheter/2012/11/30/enhorning-pa-sparen/

  9. #9 SM
    December 2, 2012

    Are they allowed to use citizen numbers that Martin? In Canada, for very good reason it is illegal to use people’s government-issued ID numbers as a general-purpose identifier. Allowing that, as they do south of the border, would make it too easy for companies to surveil people and combine databases without people’s consent; it is also dangerous because people tend to assume that such a number must be strong ID, and if they give it out widely it is easy to steal. So while a Canadian city might have such data for taxpayers (I am not sure), it would not have it for everyone and would not be able to give that data to a private marketing company.

  10. #10 Martin R
    December 2, 2012

    Dunno SM!

  11. #11 Felix Gutt
    Upstate NY
    December 2, 2012

    You can blame the research firm (after all, they are the ones responsible for administering the survey and advising the customer on how to do it right); however, having had some experience with market research, I would guess:

    a) The survey company was given three separate lists for the surveys and wasn’t paid extra for de-dupping the lists
    b) The three surveys may be tabulated separately
    c) The customer may have insisted on doing it this way

    And unfortunately I am not a survey methodologist nor am I a statistician by education or training I cannot comment on whether or not it is actually valid to survey you once for each activity. It probably depends on the questions asked; though, one would think it would be better to ask them all in one survey rather than three nigh duplicate surveys.

    However, think about a survey asking about soft drinks. They are going to ask you a series of questions about coca cola, then pepsi, then RC Cola, then whatever else is next. The questions are going to be identical but the product is going to be different. This is probably analogous to sending you one survey for each activity.

  12. #12 Martin R
    December 2, 2012

    Trouble is, polling a person changes that person. My replies to a questionnaire will be different if it’s the third identical one that day than if there’s only one.

The site is currently under maintenance and will be back shortly. New comments have been disabled during this time, please check back soon.