ArcGIS Survey123

Ethical considerations for surveys

Have you ever taken a survey that made you feel confused, or even frustrated? Or have you given a survey that respondents ghosted halfway?

Survey participation is a crucial part of success with surveys. Yet survey design experts cite sensitivity to the information provided as a common reason for non-response. Survey ethics is a way to increase trust by being sensitive to information which respondents shareFor example, ensuring respondent privacy and confidentiality can help build trust with your respondents. By incorporating ethics in both recruitment and design, completion rates can improve while helping everyone do the right thing.

First, do no harm

A good place to start are the Human Subjects Research requirements outlined for federally-funded research, which are followed by many other research organizations. At their core, these guidelines can be paraphrased as two golden rules: 

  1. The benefits/knowledge to be gained must outweigh risks (and adequacy of protection against risks). For example, ensuring respondent privacy and confidentiality can help build trust with your respondents. 
  2. Survey respondents should be informed of risks and must voluntarily consent to participating. For example, ensuring respondent privacy and confidentiality can help build trust with your respondents. Informed consent can also entail what data is being collected, stored, and used.

In other words, survey participation should be voluntary and there should be a stated benefit for the respondents. Other professional organizations have their own ethics guidelines, such as the National Association of the Practice of Anthropology, and the American Association of Geographers. 

Recruiting respondents

Recruiting voluntary participants can be challenging, but not impossible. The first step in ethical surveying is making sure that you are reaching out to the people who can truly benefit. Some issues to carefully consider are:

Survey design

Another important aspect of ethical surveying is ensuring the validity of the survey design. Wording and ordering of questions is important. Some examples of how this can go wrong are listed below:  

Asking sensitive questions

Some factors that can undermine question sensitivity are social desirability bias, or virtue signaling. For example, when asking about gambling habits or activities, it may trigger defensive responses such as “Of course I don’t gamble!” or “Me, smoke? Goodness no!” There are ways to ask these types of questions that improves the overall response rate, and the quality of responses such as: 

Configure these survey questions

Step 1. Add a Single Choice question. Then the configuration options will appear.

Step 2. Type in the direct, timebound question in the Label, and any specific definitions in the Hint. Be sure to include a zero/none option.

Question editor in Survey123 for Single Choice questions.

Notice I listed the choices in decreasing order. I did this deliberately so as to prime people to see 12+ times as the initial point of reference, and subconsciously think, “oh, I guess I don’t gamble that much, I can answer honestly.”

I can preview my new question in my survey:

Preview of how this question will appear in the survey. Label (main question) appears in large and bold font, whereas Hint (more information) appears in smaller font.

You’ll notice that there is no red star at the end of this question, indicating that it is optional (not required). This means that my respondents will be able to advance to the next question without submitting an answer should they choose to do so.

Order randomization

Similar to question order affecting responses, choice order can also bias results, as people tend to choose the first choice they agree with. If you have a survey question with choices that are not in a natural order, for instance, a non-ordered list of options about summer activities, consider randomizing the question options. When creating your survey, simply toggle on the “Show choices in random order” option, and you will see an icon of two intersecting arrows appear at the top of your question.

Single Choice configuration panel with Show Choices in Random Order toggled on.

Pro tip: Consider doing a pilot test of your question in which half the respondents get a randomized list of choices, and half get the list in the same order. Compare the results and see if people in the non-randomized group are more likely to choose options higher up in the list.

Presenting results while protecting privacy

Privacy and confidentiality requirements are very real, because geography is possibly the easiest way to identify someone. Even if someone’s name or address is not available, individuals might be able to be identified if they have unique responses to the survey, especially when responses to multiple survey questions are combined. For example, a certain neighborhood might only have one same-sex couple household with a child, only one widow who drives an electric vehicle, or only one 25-year-old female who is divorced. Depending on the survey topics and questions, I might be able to learn all about that person’s employment and earnings, medical information, and more.

When presenting your survey results, you may have to suppress low counts so that viewers of your public data and maps only see something like “3 or fewer” rather than the specific number. Another option could be to aggregate your survey points up to coarser levels of geography so as to protect respondents’ confidentiality, such as census tracts instead of block groups, or coarser categories, such as ages 18 to 34 rather than 18 to 24.

Some data visualizations in Survey123 are only enabled once a specific number of respondents complete the survey, such as the Word Cloud. A Word Cloud is a great way to visualize responses to open-ended/qualitative survey items, and requires a minimum of 20 respondents to appear.

If your respondents are likely to have an ArcGIS Online account, consider including a note to encourage respondents to sign out of ArcGIS Online before starting the survey to ensure anonymity, as Survey123 automatically captures information from ArcGIS Online accounts.

Additional resources for learning more

Survey123 resources

Have you ran a survey in which you incorporated some of these considerations? Share your work, or browse through the work of others, and exchange questions and ideas with the Survey123 user community.

Survey123 Learn Paths:

  1. Try Survey123
  2. Enter data faster and more accurately
  3. Access data results
  4. Go beyond point data

Qualitative GIS resources

About the authors

(she/her/hers) Diana loves working with data. She has over 15 years experience as a practitioner of demography, sociology, economics, policy analysis, and GIS. Diana holds a BA in quantitative economics and an MA in applied demography. She is a senior GIS engineer on ArcGIS Living Atlas of the World's Policy Maps team. Diana enjoys strong coffee and clean datasets, usually simultaneously.

Jennifer is passionate about doing in-depth user experience research to design beautiful urban experiences. She believes all technology is truly about unlocking our potential both as humans and as a community, and the first step is doing rich research that yields deep insight into both the users/customers/humans and the tools they need.

Notify of
Inline Feedbacks
View all comments

Next Article

New version of the ArcGIS Pro Performance Assessment Tool (PAT) is now available!

Read this article