PSID CALENDAR METHODS STUDY

The PSID Calendar Methods Study was a one-year project funded by the National Science Foundation (SBR-9730297). The primary goal of the project was to compare the data quality of retrospective reports on key social and economic variables with traditional standardized question-list (Q-list) interviewing methods versus those provided by event history calendar interviewing (EHC) methods. Research has shown that the retrospective reporting of economic events that had occurred at retention intervals of one year or more yield poor quality with Q-list methods (Dugoni, Lee, and Tourangeau 1996; Duncan and Mathiowetz 1985; Mathiowetz 1986; Mathiowetz and Duncan 1988). On the other hand, the quality of reporting with EHC methods on social and economic events is quite good for retention intervals of 3 to 5 years (Caspi, Moffitt, Thornton, Freedman, Amell, Harrington, Smeijers, and Silva, 1996; Freedman, Thornton, Camburn, Alwin, and Young-DeMarco, 1988). Since in this research the Q-list and EHC methods were administered with different economic items and different lengths of reference period, direct comparisons between methods is not possible. The PSID Calendar Methods Study is the first direct experimental comparison between EHC and standardized state-of-the-art Q-list interviewing methodologies. A full report of the PSID Calendar Methods Study is provided by Belli, Shay, and Stafford (2000).

Differences in retrospective reporting quality between EHC and Q-list methods is expected from cognitive theory. EHCs acquire their ability to elicit high quality retrospective reports from tapping into available idiosyncratic structures in autobiographical memory (Belli, 1998). EHCs encourage a narrative style of interviewing and remembering in which events from respondents' past experiences are used as cues to facilitate the recall of other related events. There are three types of mechanisms that are afforded within the structure of autobiographical memory: top-down, sequential, and parallel cueing. These cueing mechanisms activate interrelationships among autobiographical memories for lifetime periods, general events, and specific events, which are organized within a hierarchical structure (Barsalou, 1988; Conway, 1996). Top-down cueing occurs when memories of lifetime periods remind individuals of general events, which in turn index specific events. For example, a person remembering the lifetime period when being an assistant professor at a particular university would also have memories of specific courses taught, and memories of specific interactions with students while teaching. Sequential cueing refers to the chronological sequencing of general or specific events and their transitions within particular lifetime periods (e.g., within one’s employment history), and relies on events being organized in memory on the basis of what happened earlier (backward chronologically) and later (forward chronologically) in time. Finally, parallel cueing refers to the associations of contemporaneous events that exist across different lifetime periods. For example, changes in employment may affect, and be related to, aspects of one’s relationships with significant others and family, and vice-versa. Whereas EHCs have the potential to utilize all three types of cueing mechanisms, traditional standardized state-of-the-art Q-list surveys, primarily because they do not effectively tap into the temporal association among events, are typically limited to administering top-down cues or a limited selection of sequential ones.

Method

Interviews were collected in 1998 via telephone on paper with a random subset of respondents and questions from the regular 1997 PSID core survey. Two conditions were tested, a Q-list condition (n = 307; 84.1% response rate), and an EHC condition (n = 309; 84.4% response rate). Respondents and 20 interviewers were randomly assigned to conditions. All interviewers received general interviewer training plus 15 hours (5 hours for each of three days) of training in their respective methods. The Q-list condition used a traditional standardized survey instrument based on the core PSID questionnaire, a 25-page paper questionnaire with scripted question sequences. Interviewers were asked to optimize retrospective reporting within the constraints of standardized interviewing techniques. The EHC condition employed paper abstracts from the Q-list method as reference material, but the survey instrument, per se, was one 18"x 28" page; flexibility in using sequential and parallel cues were facilitated by the layout of timelines within domains in the instrument. With the EHC, interviewers were given scripted questions to introduce each domain and were instructed on how to use top-down, sequential, and parallel cues. Interviews were conducted in 1998 about events that occurred during 1996 and 1997.

Results

Using data from respondents collected one year earlier on events reported during the 1997 core PSID as a gold-standard, the quality of retrospective reports on 1996 events from the 1998 administration of EHC and Q-list interviews was assessed. The results of the experiment supported the notion that EHC methods promote a better utilization of beneficial autobiographical memory processes in comparison to state-of-the-art standardized Q-list methods. Better quality retrospective reporting for residence changes, number of jobs, earned income, weeks unemployed, weeks away from work due to personal illness and illness of another, was found with the EHC interviews for retrospective reports that targeted a reference period that was one to two years ago. There were no substantive differences in interviewing time between methods. Follow-up questionnaires revealed that interviewers enjoyed the EHC interviews more than the Q-list ones, they found the EHC easier to administer, and that they believed that the respondents found the questions on the EHC to be easier to answer. Despite the EHC collecting much more detailed information than the Q-list, respondents reported the same levels of burden for both methods.

The benefits of EHC interviewing in comparison to Q-list methods is illustrated by examining the marginal results of reports for residential moves and number of jobs treated categorically. Consider the proportion of respondents who reported moving at least once during 1996 or early 1997, and the proportions who reported having 0, 1, or 2 or more jobs during 1996, when responding in the 1997 core PSID gold-standard interviews and in the 1998 experimental interviews. Figure 1 depicts the results for reported moves, using two different gold-standard assessments. With the yes/no gold-standard, responses of "yes" or "no" to a question on the 1997 core PSID interview on whether respondents reported having moved since the spring of 1996 was used. With the concurrent reports of address as a gold-standard, the addresses that were concurrently reported by respondents during the 1996 and 1997 regular annual interviews were used; if the addresses were the same, the respondent was determined to not have moved during the overlap period, and if they were not the same, the respondent was determined to have moved. Figure 2 depicts the proportions of respondents on reports of number of jobs. Both figures illustrate the overall tendency in the experimental conditions to underreport relative to the gold-standard reports. The figures also show that this tendency to underreport is less pronounced in the EHC condition relative to the Q-list condition.

With reported moves (see Figure 1), and the yes/no responses as gold-standard, the Q-list condition underreported moves by a level of 6.2% (from 18.6% in the gold-standard to 12.4% in the Q-list) whereas the EHC condition only led to a 1.6% level of underreporting (from 20.5% to 18.9%). Using the concurrent address reports as a gold-standard, the data pattern with moves is replicated, with the Q-list condition leading to a 8.1% reduction in reported moves (from 20.7% to 12.6%), and the EHC condition resulting in a 4.3% reduction (from 23.0% to 18.7%).

FIGURE 1

In reporting the number of jobs held during 1996, the Q-list condition underreported the proportion of respondents who reported having held two or more jobs in 1996 by 9.4% (from 24.4% to 15.0%; see Figure 2), and overreported the proportion of respondents who held one job in 1996 by 8.2% (from 51.8% to 60.0%). Overall, then, there is a disproportionate reduction in the number of jobs reported in the Q-list condition relative to the gold-standard. These results contrast sharply with the EHC condition, which reproduced the proportions found in the gold-standard for reports of 0, 1, and 2 or more jobs, almost exactly. This finding is especially crucial given the focus of PSID on labor economics.

FIGURE 2

Figure 3 illustrates the substantive degree of differences in the correlations between experimental and gold-standard reports that were found for earned income, weeks unemployed, weeks away from work due to personal illness, and weeks away from work due to illness of another. All of these measures differed significantly between conditions in the size of Pearson correlations. Whereas the correlations between the EHC and the gold-standard varied between r = 0.6 and r = 0.9, the correlations with the Q-list and the gold-standard varied between r = 0.15 and r = 0.75. The advantages that the EHC condition reveal in comparison to Q-list interviewing are even more remarkable when considering that the format of interviewing in the EHC condition was less similar to the gold-standard than that engendered in the Q-list conditions, as the gold-standard itself was derived from a Q-list interviewing methodology.

 

FIGURE 3

Computer Assisted Interviewing EHC

The paper and pencil administration of the EHC did lead to certain disadvantages. Data entry time was significantly longer in the EHC than the Q-list condition due to the transcription of complex EHC timeline information into a usable computer data file being more cumbersome than the transcription of pre-arranged data entry fields that are available with Q-list paper instruments. In addition, although a qualitative analysis of the verbal content of 5 audiotaped EHC interviews confirmed the use by interviewers of top-down, sequential, and parallel probes, there were also noted failures of interviewers using probes optimally. For example, one interviewer missed an opportunity to engage in parallel probing between the residence and employment timelines, which led to an asynchronization in the timing of a cross-country move that corresponded to a change in employers. Thus, paper and pencil EHCs are problematic because of their reliance on the uneven skills of interviewers to recognize and administer probes in optimal situations.

A prototype computer assisted interviewing EHC (CAI-EHC), modeled from the paper EHC used in the PSID Calendar Methods Study, has been designed for purposes of solving the disadvantages associated with the costs of data entry and uneven interviewer skills found with paper EHCs (Belli, 2000; Belli, Shay, & Stafford, 1999). This prototype is available for examination on this web site. With the CAI-EHC, data files can be created or updated at the same time that the data is being collected during the interview. Regarding interviewer skills, automation can ensure the consistency and completeness of collected data, and assist interviewers to recognize situations in which probes would assist autobiographical recall.

As DOS based applications are limiting even within question-list designs (Couper, 1999), the apt representation of calendar timelines in a computer application is dependent on the advent of Windows based applications. Thus, in the CAI-EHC prototype, Microsoft Windows is used as the operating system, the screen displays are programmed in Microsoft Visual Basic, and Microsoft Access supports data entry and storage. Another critical design issue, shared with CAI Q-list applications (Couper, 1999), is to represent the same amount of information that can be addressed in paper instruments on the limited screen real estate of a CAI application, while maintaining the ability to facilitate autobiographical recall.

Reflecting the structure of autobiographical memory, the CAI-EHC prototype consists of hierarchically organized timeline domains, including domains for landmark events, residence, employment, unemployment, and time away from work. This hierarchical organization is represented by tabs that are presented in a left to right order. The reference period of data collection is for two calendar years (1996 and 1997), with the smallest unit of analysis the third of a month (first third, middle third, last third) in order to aggregate timeline information into annual weeks per year (e.g., weeks unemployed during 1996). Seasons and months are also represented on the calendar for their potential use as cues to facilitate the remembering of when events occurred for those instances in which respondents express uncertainty (e.g., "Did you start looking for work in the summer or fall of 1997? Would you say it was September, October, or November that you started looking? Was it during the beginning, end, or sometime in the middle of September that you started to look for work?").

Each domain is represented by its own screen layout, with each layout split into three sections. For each domain, the middle section is reserved for timeline data entry in that domain. The top section is usually reserved for timeline summaries from data collected in those domains that had previously been covered in the interview for purposes of parallel probing and retrieval. By pointing the cursor to a specific transition point represented in these summary timelines, the specific transitional event is displayed in a field as screen tip text. Using both the summary and data entry timelines, the interviewer can directly ask the respondent about the timing of reported events across domains, facilitating autobiographical recall through parallel retrieval mechanisms. The bottom section serves the role of providing introductory scripts to each domain. A second role for the bottom section is to present textual fields for the entry of data elements that pertain to each event spell. The timeline data entry fields and the textual fields are integrated, so that elements entered in one are represented in the other.

The CAI-EHC can assist toward the avoidance of contradictory information. As an example, any overlap of employment and unemployment spells is highlighted in red, indicating the contradiction. Dialog boxes can also be introduced specifically prompting the interviewer to seek a resolution of any inconsistencies with the respondent.

In addition to parallel retrieval processes being encouraged by the cross-referencing of data entry spells in the middle section of the screen display with summary transition events in the top section, sequential and top-down retrieval processes are also encouraged within the CAI-EHC design. By itself, requesting information for a period of time, such as respondents’ residences and labor history during 1996 and 1997, encourages sequential retrieval processes with the remembering of which events happened earlier and later in time. The CAI-EHC further encourages sequential (and parallel) retrieval processes by requiring that reported information be exhaustive for the two-year reference period (i.e., no gaps in residence, and that together, being employed, unemployed, and out of the labor force exhaust the reference period), and by the avoidance of inconsistent information as noted above. Gaps and inconsistent reports can be queried by interviewers through both sequential and parallel probing. Finally, top-down retrieval is encouraged by the textual fields displayed in the bottom section of the screen display, in which specific data elements are recorded for each spell. This feature is especially applicable in the employment domain, in which for each employer, data elements such as position and pay rate are recorded.

A CAI-EHC will be incorporated in the Panel Study on Income Dynamics (PSID) in 2003. Also ongoing at the University of Michigan is the development and testing of a CAI-EHC with a lifetime reference period that will be evaluated for its potential adoption for PSID and the Health and Retirement Study baseline interviews. As with Q-list CAI interviewing designs (Couper, 1999), additional testing and experience are expected to improve the interviewer usability of CAI-EHC instruments and their ability to produce high quality retrospective reports.

References

Barsalou, Lawrence W. 1988. "The Content and Organization of Autobiographical Memories." In Remembering Reconsidered: Ecological and Traditional Approaches to the Study of Memory, eds. Ulric Neisser and Eugene Winograd, pp.193-243. New York: Cambridge University Press.

Belli, Robert F. 1998. "The Structure of Autobiographical Memory and the Event History Calendar: Potential Improvements in the Quality of Retrospective Reports in Surveys." Memory 6:383-406.

Belli, Robert F. 2000. "Computerized Event History Calendar Methods: Facilitating Autobiographical Recall." Paper presented at the annual meeting of the American Statistical Association, Indianapolis.

Belli, Robert F., William Shay, and Frank Stafford. 1999. "Computerized Event History Calendar Methods: A Demonstration of Features, Functions and Flexibility." Paper presented at the annual meeting of the American Association for Public Opinion Research, St. Pete Beach, FL.

 

Belli, Robert F., William L. Shay, and Frank P. Stafford. 2000. "Event history calendar and question list surveys: A direct comparison of interviewing methods." Survey Methodology Program Working Paper Series, Survey Research Center, University of Michigan.

Caspi, Avshalom., Terrie E. Moffitt, Arland Thornton, Deborah Freedman, James W. Amell, Honalee Harrington, Judith Smeijers, and Phil A. Silva. 1996. "The Life History Calendar: A Research and Clinical Assessment Method for Collecting Retrospective Event-History Data. International Journal of Methods in Psychiatric Research 6:101-14.

Conway, Martin A. 1996. "Autobiographical Knowledge and Autobiographical Memories." In Remembering Our Past: Studies in Autobiographical Memory, ed. David C. Rubin, pp. 67-93. New York: Cambridge University Press.

Couper, Mick P. 1999. "The application of cognitive science to computer assisted interviewing." In M.G. Sirken, D.J. Herrmann, S. Schecter, N. Schwarz, J.M. Tanur, & R. Tourangeau (Eds.), Cognition and Survey Research (pp. 277-300). New York: Wiley.

Dugoni, Bernard, Lisa Lee, and Roger Tourangeau. 1996. "Report on the NLS/Y Round 16 Recall Experiment." Unpublished manuscript: National Opinion Research Center.

Duncan, Greg J., and Nancy A. Mathiowetz. 1985. A Validation Study of Economic Survey Data. Ann Arbor, MI: University of Michigan.

Freedman, Deborah, Arland Thornton, Donald Camburn, Duane Alwin, and LindaYoung-DeMarco. 1988. "The Life History Calendar: A Technique for Collecting Retrospective Data." In Sociological Methodology (Vol. 18), ed. C.C. Clogg, pp. 37-68. San Francisco; Jossey-Bass.

Mathiowetz, Nancy. 1986. "The Problem of Omissions and Telescoping Error: New Evidence from a Study of Unemployment." Proceedings of the Section on Survey Research Methods, American Statistical Association.

Mathiowetz, Nancy A., and Greg J. Duncan. 1988. "Out of Work, Out of Mind: Response Errors in Retrospective Reports of Unemployment." Journal of Business and Economic Statistics 6:221-29.