In the latest instalment of the NSRN/SSNB Methods series, sociologist Ryan T. Cragun considers bad, better and best ways of asking interview questions about religious affiliation and belief.

There isn’t a perfect way to ask people about their religious (non)affiliation or their (non)belief in a god or higher power. But there are wrong ways, and there are better ways. In this blog post, I don’t suggest that there is one right way to ask about these issues, but want to raise the issue of variations in question wording so those studying the nonreligious and nonbelievers are aware of the ramifications that result from subtle differences in methodology.

How Not To Ask Questions

 

One of the most instructive ways to think about good questions about nonreligious identity, non-affiliation and non-theism is to understand bad questions – or how not to ask about these things. Everyone is susceptible: the following example comes from WIN/Gallup International, which bills itself as ‘the world’s leading association in market research and polling’.[i] But even these ‘leading’ market researchers and pollsters make some of the most fundamental errors in researching and reporting nonreligion.[ii]

Here is how WIN/Gallup International asks about religious affiliation:

Irrespective of whether you attend a place of worship or not would you say you are: a. a religious person, b. not a religious person, c. a convinced atheist, d. do not know/no response.

The problems with this question are legion, but they illustrate some commonplace issues in survey research in this field.

Firstly, the researchers fail to realize that there is a difference between ‘believing’ and ‘belonging’. Religious affiliation (or ‘belonging’) has to do with an individual’s sense of connectedness or alignment with a religious (or nonreligious) tradition. Belief in a god or higher power can be and often is correlated with religious affiliation, but it is not a religious affiliation. Belonging (e.g. are you a religious person) and believing (e.g. are you a convinced atheist) are confused here.

Related to this, the question also falls foul of warnings against double-barrelled questions, or questions that ask about two separate and distinct issues simultaneously; these are strongly discouraged by principles of good survey design.[iii] Another example of a double-barrelled question would be something like, ‘Please choose yes or no for the following question, ‘Are you a Christian and a good person?’ The respondent could be a Christian and a bad person. Ze[iv] could be a good person and not a Christian. But ze could also be both or neither. The problem is that this is really two separate questions.  WIN/Gallup’s question is double-barrelled as it asks about both believing and belonging. It only adds insult to injury that the question is only double-barrelled for those who do not believe, because belief or ‘convinced theism’ is not an option  – something that is methodologically dubious, and likely to have the effect of minimizing the number of people who indicate they are atheist and/or nonreligious.

The question also violates the principle of exclusivity, that is, that each respondent should only fit into one category; respondents cannot choose ‘not a religious person’ and ‘a convinced atheist’, even though most atheists are also not religious. The effect of this is, again, to reduce the number of people in the nonreligious and atheist categories. The question also violates the principle of inclusivity, that is, that there should be a category for everyone. What if a respondent is an agnostic who visits random religious organizations, but only on holidays? Where would that person fit?  Or what about the increasingly common, ‘spiritual but not religious’?  Where do they fit?

Finally, the question also uses a modifier on ‘atheist’ that reduces the odds of people choosing that option, but there is not a comparable modifier for being a religious person (e.g., ‘a convinced religious person’).

In short, this question is a model case of how not to ask questions about religious believing and belonging. It does everything wrong.

How to Ask Better Questions

 

So that’s how not to ask questions. How can we do better? For one, a general takeaway from the WIN/Gallup question is that researchers (and others) can build specific arguments about trends in believing and belonging (not to mention behaving) by choosing carefully how they word the questions they ask.

The details of the wording matter too. The following survey questions, all from major US surveys, illustrate this, since each asks questions about religious affiliation and belief in a higher power or god in different ways, and each gets slightly different results.

Since 1972, the General Social Survey (GSS) has asked participants the following question to capture their religious affiliation:

What is your religious preference? Is it Protestant, Catholic, Jewish, some other religion, or no religion?

Pew’s question (specifically from 2014) is slightly different:

What is your present religion, if any? Are you protestant, Roman Catholic, Mormon, Orthodox such as Greek or Russian Orthodox, Jewish, Muslim, Buddhist, Hindu, atheist, agnostic, something else, or nothing in particular?’[v]

Finally, Gallup also uses different wording:

What is your religious preference – Protestant, Roman Catholic, Jewish, another religion, or no religion?[vi]

The resulting portraits of religion affiliation in the US for 2014 are shown in Figure 1.

While there are differences, most are not that large. Gallup appears to be under-representing the nonreligious, estimating that group at just 16%, while the other two surveys estimate they are above 20% of the population, a difference of potentially tens of millions of Americans. The other sizable difference is the varied percentage of Protestants in the three samples, though most of this difference is accounted for in the percentage of Christians in Gallup. Data from years prior to 2014 (and since) suggest that either the way Gallup asks about religious affiliation or Gallup’s sampling methodology is resulting in lower estimates of the percentage of nonreligious Americans than is the case in the other two surveys. Of note, all of these surveys now include the option of ‘nonreligious’ or the less preferable ‘nothing in particular’.[vii]  This was not always the case.  As late as 2005, Gallup did not include this option and in surveys prior to the change, their estimate of the percentage of the US population that was nonreligious was proportionately even lower compared to the GSS and other national surveys than it is now. More accurate estimates of the nonreligious appear to require the inclusion of this option in the question.

There are bigger differences when it comes to belief in a god or higher power. The GSS presents respondents with a variety of response options and asks them:

Please look at this card and tell me which statement comes closest to expressing what you believe about God.

• I don’t believe in God,
• I don’t know whether there is a God and I don’t believe there is any way to find out
• I don’t believe in a personal God, but I do believe in a Higher Power of some kind
• I find myself believing in God some of the time, but not at others
• While I have doubts, I feel that I do believe in God
• I know God really exists and I have no doubts about it

Pew and Gallup take a different approach on this topic. Gallup uses a single question (though with a variant for one half of the respondents):

Do you believe in God? OR Do you believe in God or a universal spirit?[viii]

Pew uses a Yes/No question initially:

Do you believe in God or a universal spirit?

It then follows up to clarify, asking participants:

How certain are you about this belief? Are you absolutely certain, fairly certain, not too certain, or not at all certain?

I show the results in Figure 2 below, having done my best to overlay Pew’s and Gallup’s response options with those of the GSS.

As Figure 2 indicates, how you ask the question matters a great deal. If you ask only a yes/no question, like Gallup does, it appears as though 86% of Americans believe in a god or higher power. That is a full 20% higher than what Pew and the GSS estimate.

Even Pew, with its two part approach, reduces the possible response options as it groups belief in a ‘God’ with belief in ‘a universal spirit’— not a small oversight, as the GSS estimates that 12% of adults in the US believe not in a personal God but in a Higher Power of some kind.

The dichotomous response options in Pew and Gallup also hide a considerable amount of nuance that is revealed in the GSS around agnosticism. The GSS estimates that 6% of Americans are agnostics; technically, Gallup and Pew do not capture agnostics, though I grouped Pew respondents who indicated their certainty of belief in a god or universal spirit as ‘not at all certain’. It’s not a perfect match, but that is precisely the point – Gallup and Pew are not actually capturing agnosticism in a clear way.

In general, comparing these instruments shows that overly general questions can be misleading. Detail from the GSS suggests that Gallup’s simple ‘yes/no’ question actually inflates the number of people who believe in a god or higher power by forcing those who doubt to opt into belief, and forcing those who truly aren’t sure if they believe to indicate they do not.

If, instead, a survey researcher wants to illustrate the nuance that exists in belief in gods or supernatural powers, they should opt for questions with multiple response options. It might even be helpful to add even more options within the ‘I know God exists’ category, such as ‘I believe in God, but no longer think God is active in the world,’ ‘I believe in a god, but it’s complicated,’ ‘I believe in many gods,’ etc.[ix]

This blog has focused on a limited set of US-based surveys, but similar effects have been found elsewhere.[x] What this work shows is that scholars interested in studying nonreligious and nonbelieving individuals need to recognize that how questions are framed will influence the results they get. There are many ways to ask about religious belonging and believing, and most of these aren’t intrinsically wrong or right – so long as the data are analysed and interpreted in a way that is sensitive to what the question is doing. On the other hand, some questions are very, very wrong, and scholars should be at pains to avoid them.

---

Ryan Cragun is an associate professor of sociology at The University of Tampa. His research focuses on Mormonism and the nonreligious and has been published in numerous professional journals.  He is also the author of several books.

---

[i]           http://www.wingia.com/en/news/losing_our_religion_two_thirds_of_people_still_claim_to_be_religious/290/

[ii]           Of note, the same question wording is used in the World Values Survey.  For an extended discussion of how question wording can affect perceived levels of religiosity in a population, see: Bruce, Steve. 2002. God Is Dead: Secularization in the West. London: Blackwell Publishers.

[iii]          Converse, Jean M. 1986. Survey Questions: Handcrafting the Standardized Questionnaire. Beverly Hills: Sage Publications.

[iv]          ‘Ze’ is a gender neutral pronoun, along with ‘zer,’ which is the possessive version of the same.

[v]           Note that Pew conflates belonging (religious affiliation) with belief as well, as they include atheist and agnostic among the options, even though those are not religious affiliations.

[vi]          Note that Gallup’s question has changed substantially over time; they added ‘no religion’ as an option in the question starting in 2006. See: http://www.gallup.com/poll/1690/religion.aspx

[vii]         Phrasing nonreligion as “nothing in particular” makes it seem as though this is just a casual and fleeting identity that is of little importance.  While that may certainly be true for some, for others, exiting religion is a monumental struggle (see: Cottee, Simon. 2015. The Apostates: When Muslims Leave Islam. London: Hurst.).

[viii]         http://www.gallup.com/poll/1690/religion.aspx

[ix]          Lois Lee recommends similar ways of extending survey options in her excellent article from 2014: Lee, Lois. 2014. “Secular or Nonreligious? Investigating and Interpreting Generic ‘not Religious’ Categories and Populations.” Religion 44(3):466–82.

[x]           On discrepancies between census and other survey data in the UK, see Voas, David and Abby Day. 2007. Secularity in Great Britain. In B. A. Kosmin and A. Keysar (eds). Secularism and Secularity: Contemporary International Perspectives. Hartford, CA: ISSSC, 95–110, and Day, Abby. 2011. Believing in Belonging: Belief and Social Identity in the Modern World. Oxford: Oxford University Press. 2011; for a detailed analysis of the effects of question wording, also in the UK case, see Field, Clive. 2014. Measuring religious affiliation in Great Britain: the 2011 census in historical and methodological context. Religion 44 (3): 357–82. DOI:10.1080/0048721X.2014.903643, and other contributions to Day, Abby and Lois Lee. 2014. Making Sense of Surveys and Censuses: Issues in Religious  Self-Identification. Religion 44 (3): 345–56. For discussion of the World Values Survey, see Egbert Ribberink, Peter Achterberg, and Dick Houtman’s earlier contribution to this blog series, Measuring Atheism: Differentiating Non-religiosity and Anti-religiosity.