Scales' Evaluative Dimension
Agree–disagree scales can be used to evaluate the level of agreement or disagreement towards a statement or a stimulus.
Item-specific scales can be used to measure variables, for which the scale options directly refer to the theoretical concept under evaluation.
Item-specific scales can be used to measure variables, for which the scale options directly refer to the theoretical concept under evaluation.
Theoretical arguments
- AD scales are clearer to interpret than vague or closed-range quantifier scales (Brown 2004).*
- People simply choose "to agree" because it seems like the commanded and polite action to take (Krosnick 1999).*
- To eliminate acquiescence avoid AD scales (Krosnick et al. 2005).*
- AD scales are more difficult to understand and make it more difficult to map the appropriate judgement (Kunz 2015).*
- AD results in more acquiescence because of its usual presentation in batteries (Saris et al. 2010).*
- AD is simpler to conduct (Schaeffer and Presser 2003).*
Empirical evidence on data quality
*DeCastellarnau, A. Qual Quant (2018) 52: 1523. doi: 10.1007/s11135-017-0533-4
- The reliability of AD scales is lower compared to IS scales [Wiley-Wiley reliability] (Alwin 2007) → YES*
- Acquiescence is found in AD scales [Acquiescence bias through SEM factor](Billiet and McClendon 2000) → YES*
- AD scales lead to lower reliabilities than IS [Pearson product-moment test-rest correlations] (Krosnick 1991) → YES*
- AD scales lead to much lower quality than IS [True-score MTMM reliability and validity] (Revilla and Ochoa 2015) → YES*
- AD scales have lower quality than IS [True-score MTMM reliability and validity] (Saris and Gallhofer 2014) → YES*
- IS scales have higher quality than AD [True-score MTMM reliability and validity] (Saris et al. 2010) → YES*
*DeCastellarnau, A. Qual Quant (2018) 52: 1523. doi: 10.1007/s11135-017-0533-4
References
Alwin, D.F. (2007). Margins of Error: A Study of Reliability in Survey Measurement. Wiley, Hoboken.
Billiet, J., McClendon, M.J. (2000). Modeling acquiescence in measurement models for two balanced sets of items. Struct. Equ. Model A Multidiscip. J. 7, 608–628. DOI: 10.1207/S15328007SEM0704_5
Brown, G.T.L. (2004). Measuring attitude with positively packed self-report ratings: comparison of agreement and frequency scales. Psychol. Rep. 94, 1015–1024DOI: 10.2466/pr0.94.3.1015-1024.
Krosnick, J.A. (1991). The stability of political preferences: comparisons of symbolic and nonsymbolic attitudes. Am. J. Pol. Sci. 35, 547–576. DOI: 10.2307/2111553
Krosnick, J.A. (1999). Survey research. Annu. Rev. Psychol. 50, 537–567. DOI: 10.1146/annurev.psych.50.1.537
Krosnick, J.A., Judd, C.M., Wittenbrink, B. (2005). The measurement of attitudes. In: Albarracin, D., Johnson, B.T., Zanna, M.P. (eds.) The Handbook of Attitudes, pp. 21–78. Lawrence Erlbaum, Mahwah.
Kunz, T. (2015). Rating scales in Web surveys. A test of new drag-and-drop rating procedures. Technische Universität, Darmstadt [Ph.D. Thesis]
Revilla, M., Ochoa, C. (2015). Quality of different scales in an online survey in Mexico and Colombia. J. Polit. Lat. Am. 7, 157–177
Saris, W.E., Revilla, M., Krosnick, J.A., Shaeffer, E.M. (2010). Comparing questions with agree/disagree response options to questions with item-specific response options. Surv. Res. Methods. 4, 61–79. DOI: 10.18148/srm/2010.v4i1.2682
Saris, W.E., Gallhofer, I.N. (2014). Design, Evaluation, and Analysis of Questionnaires for Survey Research. Wiley, Hoboken
Schaeffer, N.C., Presser, S. (2003). The science of asking questions. Annu. Rev. Sociol. 29, 65–88. DOI: 10.1146/annurev.soc.29.110702.110112
Alwin, D.F. (2007). Margins of Error: A Study of Reliability in Survey Measurement. Wiley, Hoboken.
Billiet, J., McClendon, M.J. (2000). Modeling acquiescence in measurement models for two balanced sets of items. Struct. Equ. Model A Multidiscip. J. 7, 608–628. DOI: 10.1207/S15328007SEM0704_5
Brown, G.T.L. (2004). Measuring attitude with positively packed self-report ratings: comparison of agreement and frequency scales. Psychol. Rep. 94, 1015–1024DOI: 10.2466/pr0.94.3.1015-1024.
Krosnick, J.A. (1991). The stability of political preferences: comparisons of symbolic and nonsymbolic attitudes. Am. J. Pol. Sci. 35, 547–576. DOI: 10.2307/2111553
Krosnick, J.A. (1999). Survey research. Annu. Rev. Psychol. 50, 537–567. DOI: 10.1146/annurev.psych.50.1.537
Krosnick, J.A., Judd, C.M., Wittenbrink, B. (2005). The measurement of attitudes. In: Albarracin, D., Johnson, B.T., Zanna, M.P. (eds.) The Handbook of Attitudes, pp. 21–78. Lawrence Erlbaum, Mahwah.
Kunz, T. (2015). Rating scales in Web surveys. A test of new drag-and-drop rating procedures. Technische Universität, Darmstadt [Ph.D. Thesis]
Revilla, M., Ochoa, C. (2015). Quality of different scales in an online survey in Mexico and Colombia. J. Polit. Lat. Am. 7, 157–177
Saris, W.E., Revilla, M., Krosnick, J.A., Shaeffer, E.M. (2010). Comparing questions with agree/disagree response options to questions with item-specific response options. Surv. Res. Methods. 4, 61–79. DOI: 10.18148/srm/2010.v4i1.2682
Saris, W.E., Gallhofer, I.N. (2014). Design, Evaluation, and Analysis of Questionnaires for Survey Research. Wiley, Hoboken
Schaeffer, N.C., Presser, S. (2003). The science of asking questions. Annu. Rev. Sociol. 29, 65–88. DOI: 10.1146/annurev.soc.29.110702.110112
