The Split Ballot MTMM Approach
Although the classical MTMM design is effective, it has one major problem, namely, that one has to ask the respondents three times nearly the same questions. As a consequence, people can become bored and answer with less care the questions that are repeated. It is also possible that they remember what they have said before, which means that the observed responses are not independent of each other.
In order to cope with this problem there are two possible strategies: (1) to increase the time between the observations so that the previous answers cannot be remembered anymore or (2) to reduce the number of repeated observations.
The first approach has been tried in the past. It was discussed that one can after 25 minutes repeat the same questions if similar questions are asked in between (Van Meurs and Saris 1991). This will be a solution for the second measures but not for the third ones. So Scherpenzeel and Saris (1995) have used in many experiments a panel design where at each point in time only two observations of the same questions are asked. However, this approach requires a panel which is commonly not available.
The second strategy is to ask each respondent fewer questions while compensating for the “missing data by design” by collecting data from different subsamples of the population. In doing so, the designs look very similar to the frequently used splitballot experiments and hence this design is called the “splitballot MTMM design” or SBMTMM design.
In order to cope with this problem there are two possible strategies: (1) to increase the time between the observations so that the previous answers cannot be remembered anymore or (2) to reduce the number of repeated observations.
The first approach has been tried in the past. It was discussed that one can after 25 minutes repeat the same questions if similar questions are asked in between (Van Meurs and Saris 1991). This will be a solution for the second measures but not for the third ones. So Scherpenzeel and Saris (1995) have used in many experiments a panel design where at each point in time only two observations of the same questions are asked. However, this approach requires a panel which is commonly not available.
The second strategy is to ask each respondent fewer questions while compensating for the “missing data by design” by collecting data from different subsamples of the population. In doing so, the designs look very similar to the frequently used splitballot experiments and hence this design is called the “splitballot MTMM design” or SBMTMM design.
 Two groups design
Time 1 
Time 2 

Sample 1 
Method 1 
Method 3 
Sample 2 
Method 2 
Method 3 
The table below shows the groups that provide data for variances and correlations between requests using either the same or different forms (methods).
Method 1 
Method 2 
Method 3 

Method 1 
Sample 1 

Method 2 
none 
Sample 2 

Method 3 
Sample 1 
Sample 2 
Sample 1+2 
In contrast to the classical MTMM design, no correlations are obtained for method 1 and method 2 requests. They are missing by design. Otherwise all correlations in the 9×9 matrix can be obtained on the basis of one or two samples, but the data come from different samples.
 Threegroup design
Time 1 
Time 2 

Sample 1 
Method 1 
Method 2 
Sample 2 
Method 2 
Method 3 
Sample 3 
Method 3 
Method 1 
Using this design, all requested methods are treated equally: They are measured once at the first and later at a second point in time. Therefore, there are also no missing correlations in the correlation matrix, as shown in the table below.
Samples providing data for correlation:
Samples providing data for correlation:
Method 1 
Method 2 
Method 3 

Method 1 
Samples 1 and 3 

Method 2 
Sample 1 
Samples 1 and 2 

Method 3 
Sample 3 
Sample 2 
Samples 2 and 3 
Evidently, the major advantage of this approach is that all correlations can be obtained. A second advantage is that the order effects are canceled out because each measure comes once at the first position and another time at the second position in the questionnaire.
A major disadvantage, however, is that the main questionnaire has to be prepared in three different formats for the three different groups. For this reason the 2groups SB MTMM design was used in several rounds of the ESS.
A major disadvantage, however, is that the main questionnaire has to be prepared in three different formats for the three different groups. For this reason the 2groups SB MTMM design was used in several rounds of the ESS.
 Estimating and testing
The second paper indicates that the 2Group SB MTMM design used in the ESS caused considerable problems in the estimation because of nonconvergence and improper solutions so that further research has been done to develop a better estimation procedure.
References:
Saris, Satorra and Coenders (2004) A new approach for evaluating quality of measurement instruments. Sociological Methodology, 3, 311─347.
Revilla M. and W.E. Saris (2013) The Splitballot MultitraitMultimethod Approach: Implementation and Problems, Structural Equation Modeling: A Multidisciplinary Journal, 20(1); pp. 2746.
Saris, Satorra and Coenders (2004) A new approach for evaluating quality of measurement instruments. Sociological Methodology, 3, 311─347.
Revilla M. and W.E. Saris (2013) The Splitballot MultitraitMultimethod Approach: Implementation and Problems, Structural Equation Modeling: A Multidisciplinary Journal, 20(1); pp. 2746.