Order Verbal Labels
The ordering of verbal labels can be from negative (or passive)-to-positive (or active) or from positive-to-negative.
Theoretical arguments
- Responses vary depending on the order since it provides an addition source of information (Christian et al. 2007b).*
- P-N scales may tempt respondents to rush through a set of items at a faster pace (Kunz 2015).*
Empirical evidence on data quality
*DeCastellarnau, A. Qual Quant (2018) 52: 1523. doi: 10.1007/s11135-017-0533-4
[1] The mistake in the printed version is corrected in that table.
- The order of the verbal labels does not provide significant differences on responses [Response style through distribution comparison] (Christian et al. 2007b) →NO*[1]
- No primacy effect is found by varying the order of the verbal labels [Satisficing bias through distribution comparison] (Christian et al. 2009) → NO*[1]
- More positive answers (primary effect) on P-N, non-significant evidence in the N-P format [Satisficing bias through distribution comparison] (Krebs and Hoffmeyer-Zlotnik 2010) → YES*
- The order does not have a significant impact on measurement quality [True-score MTMM reliability and validity] (Saris and Gallhofer 2007) → NO*
- The order had little or no effect on validity and reliability [True-score MTMM reliability and validity] (Scherpenzeel and Saris 1997) → NO*
*DeCastellarnau, A. Qual Quant (2018) 52: 1523. doi: 10.1007/s11135-017-0533-4
[1] The mistake in the printed version is corrected in that table.
References
Christian, L.M., Dillman, D.A., Smyth, J.D. (2006). The effects of mode and format on answers to scalar questions in telephone and web surveys. In: Lepkowski, J.M., Tucker, C., Brick, M., De Leeuw, E.D., Japec, L., Lavrakas, P.J., Link, M.W.,
Sangster, R.L. (eds.). (2007). Advances in Telephone Survey Methodology, pp. 250–275. Wiley, Hoboken
Christian, L.M., Parsons, N.L., Dillman, D.A. (2009). Designing scalar questions for web surveys. Sociol. Methods Res. 37, 393–425. doi: 10.1177/0049124108330004
Krebs, D., Hoffmeyer-Zlotnik, J.H.P. (2010). Positive first or negative first? Methodology 6, 118–127. doi: 10.1027/1614-2241/a000013
Kunz, T. (2015). Rating scales in Web surveys. A test of new drag-and-drop rating procedures. Technische Universität, Darmstadt [Ph.D. Thesis]
Saris, W.E., Gallhofer, I.N. (2007). Design, Evaluation, and Analysis of Questionnaires for Survey Research. Wiley, Hoboken
Scherpenzeel, A.C., Saris, W.E. (1997). The validity and reliability of survey questions: a meta-analysis of MTMM studies. Sociol. Methods Res. 25, 341–383.
Christian, L.M., Dillman, D.A., Smyth, J.D. (2006). The effects of mode and format on answers to scalar questions in telephone and web surveys. In: Lepkowski, J.M., Tucker, C., Brick, M., De Leeuw, E.D., Japec, L., Lavrakas, P.J., Link, M.W.,
Sangster, R.L. (eds.). (2007). Advances in Telephone Survey Methodology, pp. 250–275. Wiley, Hoboken
Christian, L.M., Parsons, N.L., Dillman, D.A. (2009). Designing scalar questions for web surveys. Sociol. Methods Res. 37, 393–425. doi: 10.1177/0049124108330004
Krebs, D., Hoffmeyer-Zlotnik, J.H.P. (2010). Positive first or negative first? Methodology 6, 118–127. doi: 10.1027/1614-2241/a000013
Kunz, T. (2015). Rating scales in Web surveys. A test of new drag-and-drop rating procedures. Technische Universität, Darmstadt [Ph.D. Thesis]
Saris, W.E., Gallhofer, I.N. (2007). Design, Evaluation, and Analysis of Questionnaires for Survey Research. Wiley, Hoboken
Scherpenzeel, A.C., Saris, W.E. (1997). The validity and reliability of survey questions: a meta-analysis of MTMM studies. Sociol. Methods Res. 25, 341–383.