Split Ballot MTMM approach

The Split Ballot MTMM Approach

Although the classical MTMM design is effective, it has one major problem, namely, that one has to ask the respondents three times nearly the same questions. As a consequence, people can become bored and answer with less care the questions that are repeated. It is also possible that they remember what they have said before, which means that the observed responses are not independent of each other.
In order to cope with this problem there are two possible strategies: (1) to increase the time between the observations so that the previous answers cannot be remembered anymore or (2) to reduce the number of repeated observations.

The first approach has been tried in the past. It was discussed that one can after 25 minutes repeat the same questions if similar questions are asked in between (Van Meurs and Saris 1991). This will be a solution for the second measures but not for the third ones. So Scherpenzeel and Saris (1995) have used in many experiments a panel design where at each point in time only two observations of the same questions are asked. However, this approach requires a panel which is commonly not available.

The second strategy is to ask each respondent fewer questions while compensating for the “missing data by design” by collecting data from different subsamples of the population. In doing so, the designs look very similar to the frequently used split-ballot experiments and hence this design is called the “split-ballot MTMM design” or SB-MTMM design.

Two groups design

The two-group split-ballot MTMM design is structured as follows. The sample is split randomly into two groups. One group has to answer three survey items formulated in the first form while the other group is given the same survey items presented in a second form, in the MTMM literature the different forms are called “methods.” In the last part of the questionnaire all respondents are presented with the three items, which are now formulated in the method 3 format. The two-group split-ballot MTMM design can be summarized as tabulated as follows:

	Time 1	Time 2
Sample 1	Method 1	Method 3
Sample 2	Method 2	Method 3

The table below shows the groups that provide data for variances and correlations between requests using either the same or different forms (methods).

	Method 1	Method 2	Method 3
Method 1	Sample 1
Method 2	none	Sample 2
Method 3	Sample 1	Sample 2	Sample 1+2

In contrast to the classical MTMM design, no correlations are obtained for method 1 and method 2 requests. They are missing by design. Otherwise all correlations in the 9×9 matrix can be obtained on the basis of one or two samples, but the data come from different samples.

Three-group design

The three-group design proceeds as the previous design except that three groups or samples are used instead of two, leaving us with the following scheme:

	Time 1	Time 2
Sample 1	Method 1	Method 2
Sample 2	Method 2	Method 3
Sample 3	Method 3	Method 1

Using this design, all requested methods are treated equally: They are measured once at the first and later at a second point in time. Therefore, there are also no missing correlations in the correlation matrix, as shown in the table below.

Samples providing data for correlation:

	Method 1	Method 2	Method 3
Method 1	Samples 1 and 3
Method 2	Sample 1	Samples 1 and 2
Method 3	Sample 3	Sample 2	Samples 2 and 3

Evidently, the major advantage of this approach is that all correlations can be obtained. A second advantage is that the order effects are canceled out because each measure comes once at the first position and another time at the second position in the questionnaire.

A major disadvantage, however, is that the main questionnaire has to be prepared in three different formats for the three different groups. For this reason the two-groups SB-MTMM design was used in several rounds of the ESS.

Estimating and testing

Since the assignment of individuals to groups has been made at random, and there is a large sample in each group, the most natural approach for estimating is the multiple-group Maximum likelihood estimator (Jöreskog 1971). The procedure is available in most of the SEM software packages. We refer to this approach as MG-ML. A common model, the classical or true score MTMM model, is fitted across the samples, with equality constraints for all the parameters which the groups have in common. With the current software and applying the theory of multiple-group analysis, estimation can be made by using the maximum likelihood (ML) method or any other standard estimation procedure in SEM. In the case of non-normal data, robust standard errors and test statistics are available in the standard software packages. For a review of multiple-group analysis in SEM models as applied to all the designs enumerated in the present chapter, see Satorra (2000).

The second paper indicates that the two-group SB-MTMM design used in the ESS caused considerable problems in the estimation because of non-convergence and improper solutions so that further research has been done to develop a better estimation procedure.

References:
Saris, Satorra and Coenders (2004) A new approach for evaluating quality of measurement instruments. Sociological Methodology, 3, 311─347.

Revilla M. and W.E. Saris (2013) The Split-ballot Multitrait-Multimethod Approach: Implementation and Problems, Structural Equation Modeling: A Multidisciplinary Journal, 20(1); pp. 27-46.

<< the true score MTMM model

the EUPD approach >>

​The Split Ballot MTMM Approach

The Split Ballot MTMM Approach