The Survey Quality Predictor (SQP 2.0)
The Survey Quality Predictor (SQP) is:
- an extensive open-source database of survey questions and quality estimates built up through the collaboration of the users. The SQP database contains a wide range of survey questions concerning many different topics in many different forms and languages.
- a coding system of formal and linguistic characteristics of survey questions which allows a prediction of their reliability, validity and quality to be obtained. This prediction is based on a meta-analysis of the relationships between the quality estimates of survey questions obtained through Multitrait-Multimethod (MTMM) experiments and the formal and linguistic characteristics of the questions in those experiments.
- a tool for improving questions. By providing information about the quality of different question formats, the software can help design better questions.
The quality of a survey question is defined, in this context, as the strength of the relationship between the latent variable of interest and the response to the survey question, the observed variable. The quality can be computed by taking the product of reliability and validity.
The information provided by SQP is particularly useful to:
How to use this information.
The coefficients predicted by SQP are presented in the measurement model of a simple concept ($F$) below where all variables are standardized except the random error term ($e^{'}$). The method effect ($M$) and the random error term ($e^{'}$) are assumed to be independents of each other and of the concept ($F$).
The information provided by SQP is particularly useful to:
- consult, compare and evaluate questions which are part of the SQP database,
- design new questionnaires, since SQP provides suggestions of changes in question format for improving the quality of questions.
- correct for measurement error in the substantive analyses.
How to use this information.
The coefficients predicted by SQP are presented in the measurement model of a simple concept ($F$) below where all variables are standardized except the random error term ($e^{'}$). The method effect ($M$) and the random error term ($e^{'}$) are assumed to be independents of each other and of the concept ($F$).
This is a measurement model where $Y$ is the standardized observed variable, $F$ is the standardized simple concept, $T$ is the standardized true score, $M$ is the standardized method factor which affects $T$, and $e^{´}$ is the random error affecting $Y$. The coefficient $v$ represents the validity coefficient, $r$ the reliability coefficient and and $\mu$ the method effect. All coefficients are standardized coefficients.
This model can be formulated in two equations:
$T = v F + {\mu} M $
$Y = r T + e^{´}$
or by substitution
$Y = q F + m M + e^{´}$
where $q = r v$ and $ m = r {\mu}$
This gives the interpretation that $q$ represents the effect of the concept $F$ on the observed variable and that $q^{2}$ is the explained variance in $Y$ by $F$ or the quality of the measure $Y$ for $F$.
In the same way m represents the systematic effect of the method factor on the observed variable $Y$ and $m^{2}$ is equal to the explained variance in $Y$ caused by the method used.
Using this notation the following equivalent measurement model can be specified:
We have seen that SQP provides estimates of $r$ and $v$ therefore $q$ is also known.
Since $ m = r {\mu}$ and ${\mu}^{2}=1-v^{2}$ it follows that $m$ is also known if $r$ and $v$ are known for a question.
Therefore SQP can provide the basic information for the evaluation of the quality of a standardized observed variable for its concept of interest.
Given that $q^{2}$ is the explained variance in $Y$ it also follows that variance of the unstandardized variable $f$ denoted by ${\sigma}_{ff}=q^{2}{\sigma}^{2}_{yy}$ where ${\sigma}_{yy}$ is the variance of the unstandardized variable $y$.
To use the software go to sqp.upf.edu
For information about the software we refer to:
Oberski D., T. Gruner and W. Saris (2011). The prediction procedure the quality of the questions based on the present data base of questions In Saris W. , D. Oberski, M. Revilla, D. Zavala, L. Lilleoja, I.Gallhofer and T. Gruner (2011) The development of the program SQP 2.0 for the prediction of the quality of survey questions, RECSM Working paper 24 (chapter 6)
For a complete overview of the approach we refer to:
Saris W.E. and I.N. Gallhofer (Second Edition, 2014). Design, evaluation and analysis of questionnaires for survvey research. Hoboken, Wiley
Since $ m = r {\mu}$ and ${\mu}^{2}=1-v^{2}$ it follows that $m$ is also known if $r$ and $v$ are known for a question.
Therefore SQP can provide the basic information for the evaluation of the quality of a standardized observed variable for its concept of interest.
Given that $q^{2}$ is the explained variance in $Y$ it also follows that variance of the unstandardized variable $f$ denoted by ${\sigma}_{ff}=q^{2}{\sigma}^{2}_{yy}$ where ${\sigma}_{yy}$ is the variance of the unstandardized variable $y$.
To use the software go to sqp.upf.edu
For information about the software we refer to:
Oberski D., T. Gruner and W. Saris (2011). The prediction procedure the quality of the questions based on the present data base of questions In Saris W. , D. Oberski, M. Revilla, D. Zavala, L. Lilleoja, I.Gallhofer and T. Gruner (2011) The development of the program SQP 2.0 for the prediction of the quality of survey questions, RECSM Working paper 24 (chapter 6)
For a complete overview of the approach we refer to:
Saris W.E. and I.N. Gallhofer (Second Edition, 2014). Design, evaluation and analysis of questionnaires for survvey research. Hoboken, Wiley
