Point/Counterpoint Not the Time to Abandon the Food Frequency Questionnaire: Counterpoint

Virtually all epidemiologists (including us) who have investigated relationships between diet and chronic disease in the past 20 years have used food frequency questionnaires (FFQ) to assess diet and will continue to publish findings based on FFQ-derived data. However, in our editorial ‘‘Is it Time to Abandon the Food Frequency Questionnaire,’’ we addressed how we should conduct epidemiologic studies in the future. Our response to Drs. Willett and Hu is organized around the question of validity. In Psychometric Theory (1), Jum Nunnally and Ira Bernstein state that ‘‘Validity usually is a matter of degree rather than an all-or-none property, and validation is an unending process.’’ Let us look afresh at three explicit standards by which the validity of the FFQ can be judged: face validity, construct validity, and predictive validity. Face validity can be described simply as judgments by experts about the ability of a measuring instrument to function as intended. Applied to an FFQ, we can frame this question as, ‘‘Could the perceived frequency of consuming 100-125 foods or food groups over the past year, with some additional questions about food preparation and purchasing, accurately measure usual nutrient intake from the foods that were actually consumed?’’ Lacking data from a survey of experts, we must consider this question from our own perspective and that of our immediate colleagues. First, let us consider the face validity of an FFQ at the individual-item level. It seems unlikely that many nutritionists would use the estimated frequency of consuming ‘‘Beef, pork, or lamb as a sandwich or mixed dish, e.g., stew, casserole, lasagna, etc.’’ to estimate an individual’s nutrient intake from these foods. Indeed, how could one sensibly assign a single nutrient composition to this item given the enormous number of nutritionally dissimilar foods commonly consumed in the United States, Europe, or Asia which could be included? Consider such basic, broadscale dissimilarities as the meats themselves; the relative amounts of meat, vegetables, cheese, and other ingredients used in mixed dishes; and the use of fats during preparation. Many FFQ items, including even relatively simple items such as ‘‘Pizza’’ and ‘‘Tortillas,’’ raise similar issues. Second, if we consider face validity at the level of the entire FFQ, it seems unlikely that many epidemiologists or nutritionists would, after completing an FFQ, judge that their answers accurately reflected their usual dietary patterns or could be used to measure their own energy and nutrient intake. Many epidemiologists would probably agree that, in specific contexts and for specific purposes, FFQs can do a reasonable job in characterizing individuals on very broad dietary patterns. But there is little reason to believe that the FFQ could accurately capture the diversity and variability of foods actually consumed or measure an individual’s usual nutrient intake from these foods. From the perspective of face validity, it seems far more likely that dietary assessment approaches (e.g., food records) based on foods that are actually consumed will more accurately measure true nutrient intake. Construct validity can be described as the degree to which the data collected reflect or measure the variable of interest. Applied to an FFQ, we can frame this question as the magnitude of the variance shared between nutrient intake estimated from the FFQ and true nutrient intake. We do have data to evaluate this question, although obtaining them requires use of recovery biomarkers to measure true nutrient intake, which are limited currently to doubly-labeled water for total energy and 24-hour urinary nitrogen excretion for total protein. The OPEN study showed clearly that the correlations of FFQ-derived data with these recovery biomarkers are poor (2). As noted by Willett and Hu, the magnitude of these correlations is limited by the reliability of the recovery biomarkers. If we assume a conservative 0.70 reliability for these biomarkers and inflate the values reported in the OPEN study accordingly, the maximum possible correlation of an FFQ with true energy intake would be 0.14 for women and 0.28 for men, and for protein, corresponding values would be 0.43 for men and 0.46 for women. The construct validity of FFQ-based assessments of other nutrients can be less rigorously evaluated using concentration biomarkers (e.g., serum selenium or phospholipid N-3 fatty acids). These biomarkers reflect diet, but because they are also affected by other environmental and metabolic factors, it is difficult to interpret construct validity based on the range of correlations (0.2-0.6) generally observed between concentration biomarkers and FFQ-derived data. In general, we know that FFQ-derived data correlate with several biomarkers, but the amount of shared variance between measures ranges from 4% to 36%. From our perspective, the most important standard for validity is predictive validity, which can be described as the ability of an instrument to predict functional relationships between the construct measured and an observable outcome. Applied to an FFQ, this can be framed as the ability to detect associations between a dietary exposure and disease outcome. As noted by Willett and Hu, there are several outcomes that can be predicted from FFQ-derived data; for example, high consumption of fruits and vegetables predicts reduced risk of cardiovascular disease. But many associations that we would expect to observe between diet and cancer, based on an understanding of cancer biology, animal experiments, and ecologic studies, have not been detected using FFQ-derived data. One way to evaluate predictive validity is to compare, in an observational study, results using FFQ-derived data to results using data from a dietary assessment method based on capturing foods actually consumed. A second approach, based on an intervention trial, is to compare the observed experimental outcomes with those predicted by measurement of dietary change derived from alternative dietary assessment methods. We now have sufficient data to examine both of these approaches in the context of the long-hypothesized relationship between fat intake and breast cancer risk. In 2003, Bingham published results of a study of dietary fat and breast 1759