Knowledge Digraph Contribution Analysis of Protocol Data

A knowledge digraph de nes a set of semantic (or syntactic) associative relationships among propositions in a text (e.g., Graesser and Clark (1985) conceptual graph structures and the causal network analysis of Trabasso & van den Broek, 1985). This paper introduces the Knowledge Digraph Contribution (KDC) data analysis methodology for quantitatively measuring the degree to which a given knowledge digraph can account for the occurrence of speci c sequences of propositions in recall, summarization, talkaloud, and question-answering protocol data. KDC data analysis provides statistical tests for selecting the knowledge digraph which "bestts" a given data set. KDC data analysis also allows one to test hypotheses about the relative contributions of each member in a set of knowledge digraphs. The validity of speci c knowledge digraph representational assumptions may be evaluated by comparing human protocol data with protocol data generated by sampling from the KDC distribution. Speci c concrete examples involving the use of actual human recall protocol data are used to illustrate the KDC data analysis methodology. The limitations of the KDC approach are also brie y discussed. Knowledge Digraph Contribution Analysis 2 Semantic and syntactic relationships among propositions have been shown to play a crucial role in describing how people understand (e.g., Kintsch & van Dijk, 1978; Kintsch, 1988; Trabasso & Magliano, 1996), recall (e.g., Bower, Black, & Turner, 1979; Graesser, 1978; Graesser et al., 1980; Jarvella, 1971; Stein & Glenn, 1979; Trabasso & van den Broek, 1985; Varnhagen, 1991) summarize (e.g., Kintsch & van Dijk, 1978; Rumelhart, 1977; van den Broek & Trabasso, 1986) and answer questions about texts (Graesser & Clark, 1985; Graesser & Hemphill, 1991; Millis & Barker, 1996). It is usually assumed that such semantic and syntactic relationships may be represented using a pair-wise representational scheme of the form: Proposition X is related by relation R to Proposition Y. Thus, it is usually convenient to represent such pairwise relations as a directed graph or digraph. A digraph is simply a set of pair-wise relationships among a particular set of elements. In this case, the set of elements include the propositions which are explicitly mentioned within a particular text (but could also include implicitly referenced propositions as well). For example, the "conceptual graph structures" described by Graesser & Clark (1985), the "goal hierarchy" of Lichtenstein & Brewer (1980), and the "scripts" described by Bower, Black, & Turner (1979) are examples of special types of knowledge digraphs. Another important example of a knowledge digraph is the causal network described by Trabasso, van den Broek, and their colleagues (Trabasso & van den Broek, 1985; van den Broek, 1988, 1990). Such causal networks are created by examining the set of causal relationships among pairs of propositions in a given text using Mackie's (1980) counterfactual notion of causality. It has been demonstrated that statements which have more causal connections and which are located on the main causal chain are more likely to be included in story recall (Trabasso & van den Broek, 1985) and story summarization protocols (van den Broek & Trabasso, 1986). The causal network analysis has also been shown to be useful for understanding how on-line commentaries about a story are related to explicit statements in the text (e.g., Trabasso & Magliano, 1996). Knowledge digraph contribution analysis. In general, previous research concerned with the investigation of the psychological reality of particular knowledge digraphs has Knowledge Digraph Contribution Analysis 3 focussed upon only general characteristics of a proposed knowledge digraph as opposed to speci c structural features. For example, statements with more causal connections to other statements are more likely to be included in recall and summarization protocols as previously noted. True, such experiments provide support for the role of causal knowledge structures in recall and summarization protocol production, but the relationship between the speci c knowledge digraph structures and the data is not "tight". Ideally, one would like to quantitatively measure the extent to which a particular knowledge digraph contributes to the explanation of statistical regularities in the data. With such a quantitative measure, deliberate incremental progress could be made in empirically evaluating and re ning theoretically-motivated knowledge digraph analyses. For example, two researchers might generate slightly di erent causal knowledge digraphs for the same text. A conventional data analysis examining the likelihood that statements with more causal connections are included in a protocol might nd that both causal knowledge digraph analyses are about equally e ective at accounting for the data. However, such a conventional analysis has a number of obvious intrinsic limitations which could account for the lack of observed di erences. First, the sequential order in which propositions appear in the protocol is not taken into account. Thus, exactly the same results would be obtained if all subjects generated scrambled recall or summarization protocols. Second, the ne-grained relational structure of the two different causal knowledge digraphs is not taken into account. Thus, two di erent causal knowledge digraphs might make exactly the same predictions about which statements have many causal connections and which statements have fewer causal connections. A statistical analysis less susceptible to these criticisms would explicitly try to use available data regarding the ordering of propositions in a protocol in order to estimate a speci c quantitative measure of the relevance of a particular causal knowledge digraph. Thus, the statistical analysis must explicitly incorporate a mechanism for representing knowledge digraphs into its probabilistic modeling assumptions. Such a statistical analysis has been developed and re ned over the years (Golden et al., 1993; Golden, 1994; Golden, 1995; Golden, 1997), and will be referred to in this paper as Knowledge Knowledge Digraph Contribution Analysis 4 Digraph Contribution (KDC) analysis. KDC analysis is designed for the analysis of protocol data which can be represented as an ordered sequence of propositions. Examples of such protocol data include: (i) talk-aloud protocol data, (ii) recall protocol data, (iii) summarization protocol data, and (iv) question-answering protocol data. KDC analysis should be of great interest to researchers in the eld of text comprehension and memory for at least three reasons. First, the analysis can be used to decide which of several sets of knowledge digraphs "bestts" a given set of protocol data. Second, the analysis can be used to estimate the relative contributions of a given set of knowledge digraphs and examine how those relative contributions change as a function of experimental manipulations. And third, sequences of propositions can be generated by the analysis which are presumably "typical" of the protocol data with respect to a given set of knowledge digraphs. If the proposition sequences are not human-like, then such discrepancies may provide valuable insights into how to improve the knowledge digraph representations and thus model validity. The critical assumption of KDC analysis. The key underlying assumption of the KDC analysis is that: If proposition A is semantically (or syntactically) related to proposition B according to a particular pair-wise knowledge relationship in a particular knowledge digraph, then proposition B will have a tendency to follow proposition A in the protocol. A considerable amount of experimental evidence in the literature is consistent with this sort of assumption. Jarvella (1971) showed that the likelihood of recalling two adjacent clauses in a text is greater when the two clauses are part of the same sentence as opposed to the case where the two clauses are still adjacent but originated from di erent sentences. Black and Bern (1981) and Myers, Shinjo and Du y (1987) showed that if two propositions are causally related, then one of the two propositions can serve as a retrieval cue for the other proposition. In addition, text recall studies have demonstrated subjects tend to recall texts in their canonical orders as dictated by relevant underlying schemata (e.g., Bischofshausen, 1985; Bower, Black, & Turner, Knowledge Digraph Contribution Analysis 5 1978; Levorato, 1991; Lichtenstein & Brewer, 1980; Mandler, 1978; Mandler & DeForest, 1979). Since the schemata in such studies may be represented as knowledge digraphs, such text recall studies are also generally consistent with the key assumption of KDC analysis. Evidence supporting the KDC assumption for summarization protocol data is also available. Rumelhart (1977) used story grammars to make predictions about likely sequences of propositions which would best account for human-generated summaries of texts. Graesser and his colleagues (Graesser & Clark, 1985; Graesser & Franklin, 1990; Graesser & Hemphill, 1991; Millis & Barker, 1996) have used conceptual graph structures to make predictions about the likelihood that a given "answer"-type proposition would be elicited from a subject given a particular "question"-type proposition. Trabasso & Magliano (1996) have shown that a causal network analysis of a text can provide important insights into understanding the sequential order of propositions generated by subjects when they are asked to think-aloud during the process of text comprehension. Relationship to classical sequential data analysis methods. Classical sequential data analysis methods are not typically used to analyze sequential structure in protocol data. Moreover, even if such techniques were used more extensively in the analysis of sequential structure in protocol data, existing classical sequential data analysis methods (e.g., Bous eld & Bous eld, 1966; Gottmann and Roy, 1990) are not appropriate tools for investigating the issues of interest in this paper. True, such data analysis methods are similar to the approach proposed here in the sense that they allow one to investigate the likelihood that one proposition will tend to follow another proposition in a text. On the other hand, classical sequential data analysis methods do not possess mechanisms for estimating the relative contributions of a given set of knowledge digraphs towards explaining sequential regularities in human protocol data. Moreover, most classical sequential data analysis methods tend to be exploratory in nature and are not designed to test and/or con rm speci c a priori hypotheses about the data. Finally, since the complexity of the data used in KDC analysis is great but the number of free parameters is small, a KDC model is likely to be misspeci ed (i.e., wrong). Thus, special methods Knowledge Digraph Contribution Analysis 6 must be used in deriving the statistical tests so that reliable statistical inferences can be made in the presence of model misspeci cation (see Golden, 1995, 1996, for relevant reviews). Such methods are not typically used in the classical sequential data analysis literature. Connectionist representation of distributional assumptions. When describing a new statistical methodology, the underlying distributional assumptions of the statistical analysis must be described. In the case of linear regression, one assumes the dependent variable may be represented as a weighted sum of predictor variables plus a zero-mean Gaussian noise error term. Assumptions such as "normally distributed", however, are not applicable in the analysis of categorical time-series data since a proposition is viewed as a particular value of a discrete nominal-type random variable. The distributional assumptions of the KDC analysis are very straightforward and natural and have their origins in log-linear modeling methods. The assumptions in their pure equation form may be found in the Appendix. For expository reasons, an intuitive interpretation of these assumptions in the form of a connectionist network architecture is presented in the body of this paper. It is important to emphasize, however, that the KDC analysis is not a connectionist model of speci c cognitive processes involved in text comprehension and memory. Rather, the KDC analysis is a sophisticated data analysis tool which is applicable to the analysis of categorical time-series data and hence to the ordering of propositions in human subject generated protocol data. The close relationship between connectionist models of data analysis and classical methods of data analysis have been recognized by many researchers. For example, it can be shown that an analysis of variance (ANOVA) statistical analysis methodology is simply a special type of linear regression analysis. A linear regression analysis, in turn, is formally equivalent to learning the connections in a speci c linear connectionist unit and then "pruning" connection weights which are not signi cantly di erent from zero (e.g., Golden, 1996, Chapters 7,8). Overview. This paper is organized in the following manner. First, the underlying assumptions of the KDC analysis are described. The assumptions show how the probability that a particular target proposition occurs at a speci c location in the proKnowledge Digraph Contribution Analysis 7 tocol may be expressed in terms of the propositions preceding the target proposition in the protocol and a weighted sum of knowledge digraphs. Second, a method for determining the relative contribution weights of the di erent knowledge digraphs is described. Third, a new statistical test is described for deciding whether one combination of knowledge digraphs or another combination of knowledge digraphs provides a better explanation of a given set of protocol data. Fourth, new statistical tests that were speci cally derived for the KDC analysis are described. Such tests may be used for a variety of purposes including: (i) identifying the most important knowledge digraphs, (ii) comparing the relative contributions of competing knowledge digraphs, and (iii) examining how the relative contributions of a set of knowledge digraphs changes as a function of speci c experimental manipulations. Fifth, a method for sampling protocols from the KDC protocol probability distribution is discussed. The synthesized protocols obtained from sampling the KDC protocol probability distribution are useful since they can be compared qualitatively and quantitatively to human generated protocol data. This methodology is important for exploring the psychological validity of the KDC probability distribution by comparing statement recall probability statistics from human protocol data with synthesized protocol data. The sampling methodology also provides a mechanism for evaluating the KDC probability distribution and a given set of knowledge digraphs as an arti cially intelligent system capable of generating human-like responses to requests concerned with text recall, summarization, and question-answering. KDC Modeling Assumptions Protocol Data Representational Assumptions A text proposition table explicitly identi es all text propositions required for modeling a given text. The text proposition table for Stein and Glenn's (1979) Fox and Bear text is shown in Table 1. In the left-hand column of the text proposition table is the text proposition identi cation code which identi es the text proposition of interest. In the right-hand column is an illustrative "sentence fragment" instance of the text proposition concept. Knowledge Digraph Contribution Analysis 8 |||||||||||| Insert Table 1 About Here |||||||||||| Consider the following human subject generated recall protocol: A fox and a bear were hungry so they decided to get some chickens. They went to a hen house to get the chickens. At the hen house, the bear climbed upon on the hen house roof while the fox got the chickens. The fox was in the hen house when the weight of the bear caused the hen house roof to collapse. The fox and the bear were trapped in the hen house. A farmer came out to see what was the matter. This protocol might be coded as the following ordered sequence of propositions: f 4 ; f 9 ; f 12 ; f 18 ; f 24 ; f 25 ; f 26 : Notice how the statements "while the fox got the chickens" and "the fox was in the hen house" are not represented in this particular representational scheme since propositions corresponding to these concepts are not present in the text proposition table. There is no reason why such "implicit propositions" could not be added to the text proposition table. The decision to add such propositions to the text proposition table is left to the researcher who is using the KDC analysis. Knowledge Digraph Representation Assumptions Figures 1 and 2 depict two possible knowledge digraphs for the Fox and Bear text: (1) a causal network knowledge digraph (solid arrows), and (2) an episodic memory knowledge digraph (dashed arrows). The causal network knowledge digraph (Figure 1) was generated using the methodology reviewed by Van Den Broek (1990; also see Trabasso, Secco, and Van Den Broek, 1984). The episodic memory knowledge digraph (Figure 2) speci es how knowledge of the original order of propositions in the text in uences the subject's recall. For example, since proposition 6 is read by subjects Knowledge Digraph Contribution Analysis 9 immediately after they read proposition 5, there is an episodic memory link connecting node 5 to node 6 in Figure 2. The term "episodic memory" in this context refers to the concept of memory for the speci c event that describes the process of reading the text. Notice that the links to and from the propositions labeled Initial Situation (node 1) and Final Situation (node 27) correspond to a psychological theory of how the subject enters and exits a particular knowledge digraph. ||||||||| Insert Figure 1 About Here ||||||||| ||||||||| Insert Figure 2 About Here ||||||||| For long complex narratives, it may be appropriate to represent the causal knowledge digraph as a set of several smaller (possibly overlapping) causal knowledge digraphs where each smaller causal knowledge digraph corresponds to an analysis of a speci c episode in the narrative text. The KDC analysis will estimate a "contribution weight" for each knowledge digraph in the analysis. Thus, subdividing a larger causal knowledge digraph into several smaller knowledge digraphs has the advantage that individual contribution weights for each smaller knowledge digraph can be estimated. In principle, this means that one can subdivide knowledge digraphs until a contribution weight is estimated for each individual link in the knowledge digraph. The disadvantage of such a procedure, however, is that the resulting KDC probability distribution will have many more free parameters to estimate and thus larger sample sizes will be required for reliable statistical inferences. Knowledge Digraph Contribution Analysis 10 KDC Protocol Distributional Assumptions As mentioned earlier, just as one must assume that the data is normally distributed before using a linear regression analysis, there are a number of assumptions one must make before using the KDC analysis to analyze protocol data. These assumptions essentially describe how the likelihood of a given human subject's protocol is computed in terms of the knowledge digraphs de ned by the theorist. For expository reasons, these distributional assumptions are qualitatively described in terms of the connectionist network architecture shown in Figure 3. The speci c mathematical details are provided in a terse form in the Appendix. It is important to emphasize again, however, that the connectionist model described in this paper does not pretend to be a model of speci c cognitive processes involved in text comprehension or recall. Rather, the connectionist model instantiates a particular psychological theory about how the propositions in a protocol are related to one another in terms of a given set of knowledge digraphs. Connectionist modeling terminology. For the purposes of this paper, a connectionist network consists of a set of nodes or computing units. Each node in the network is associated with a real number which is referred to as the node's activation level. The activation level of one node in the network is assumed to in uence the activation level of another node. Usually the degree of this in uence is represented by another real number which is referred to as the connection weight or connection strength between the two nodes. A set of connection weights is referred to as a connectivity pattern. An introduction to connectionist networks may be found in Chapter 1 of Golden (1996). ||||||||| Insert Figure 3 About Here ||||||||| Distributional assumption 1: Number of propositions in a protocol. It is assumed that the number of propositions included in a protocol may be estimated using the average number of propositions recalled by a subject without taking into account other Knowledge Digraph Contribution Analysis 11 story-speci c factors associated with the "content" of the text. This is a strong assumption and future research will explore methods for weakening this assumption in appropriate ways. Distributional assumption 2: Local coherence assumption. A number of researchers (e.g., Fletcher, 1981, 1986; Fletcher & Bloom, 1988; Trabasso & Magliano, 1996) have shown that local coherence strategies such as the recency (Kintsch & van Dijk, 1978), leading edge (Kintsch & van Dijk, 1978), or current state (Fletcher & Bloom, 1988) strategies play important roles in on-line comprehension processes. Since a reader's understanding of the text is created as a by-product of on-line comprehension processes, it seems reasonable to expect that the existence of such local coherence strategies during the process of text comprehension will in uence the ordering of propositions in subjectgenerated protocols. In addition, one could assume that the degree to which particular propositions are currently active in the reader's working memory during the protocol generation process, plays an important role in determining the likelihood of occurrence of the next proposition which will be generated by the subject. Assumptions such as these will be referred to as local coherence assumptions since they assume that the global structural relationships among elements in the protocol can at least be partially understood in terms of speci c types of local structural relationships. Figure 3 illustrates how a recency type local coherence assumption may be instantiated in the KDC model for a proposition table consisting of only ve propositions (i.e., propositions 5; 6; 7; 8; and 9 from the Fox and Bear text). For each proposition in the proposition table, there is a corresponding working memory node. The rst proposition in the protocol (i.e., proposition 5) activates its respective working memory node only. The second proposition in the protocol (i.e., proposition 7) is then processed and the working memory node corresponding to the second proposition in the protocol is activated. However, it is assumed that the activation level of the working memory node corresponding to the rst proposition in the protocol has now partially decayed by a factor of 50% (i.e., decay rate = 0.50) in strength. The third proposition in the protocol (i.e., proposition 8) is then processed but by this time the activation levels of the working memory nodes corresponding to propositions 5 and 7 have partially Knowledge Digraph Contribution Analysis 12 decayed to activation levels (0:5) 2 and 0:5 respectively. Figure 3 shows that the state of the KDC model during the processing of third item in the protocol (proposition 8) which, in turn, results in a prediction of the probability distribution of the ve possible values for the fourth item in the protocol. Assumptions which are consistent with more sophisticated psychological theories regarding conditions for maintenance of information in working memory during comprehension and recall processes can also be readily formulated within the KDC framework. For example, depending upon the theorist's hypotheses regarding how the memorability of one proposition compares with the memorability of another proposition, some propositions may fade from working memory more rapidly than other propositions. Similarly, di erent strategies for maintaining and reducing the activation levels of working memory nodes can be developed. In particular, a distributional assumption consistent with the current-state strategy (Fletcher & Bloom, 1988) may be expressed by having working memory nodes corresponding to causal consequences inhibit working memory nodes corresponding to their respective causal antecedents. Thus, an inhibitory connection from working memory node "RUN(F & B, FARM)" to working memory node "KNOW(F& B, LOC(FARM))" would indicate that the causal antecedent "KNOW(F & B, LOC(FARM))" should not be maintained in working memory once the causal consequence "RUN(F & B, FARM)" has been activated. Distributional assumption 3: Superposition assumption. Again remembering that a protocol is an ordered sequence of propositions, let a particular proposition in a protocol be referred to as the target proposition. The superposition distributional assumption may be viewed as a simple and straightforward speci c instantiation of the idea that any combination of the set of knowledge digraphs is in uential in predicting the likelihood the target proposition will be included in the protocol. The superposition assumption also states that this likelihood is functionally dependent upon the set of knowledge digraphs, their respective contribution weights, and the activation levels of the working memory nodes. More speci cally, it is assumed that the likelihood a target proposition is included in the protocol is a weighted sum of the activation levels of the Knowledge Digraph Contribution Analysis 13 working memory nodes where the "weights" are determined by the set of knowledge digraphs and their respective contribution weights. Although KDC analysis is applicable to cases where there are many di erent types of knowledge digraphs, the following simple example will illustrate the superposition distributional assumption. In this example, only two knowledge digraphs are considered: A causal knowledge digraph and an episodic memory knowledge digraph (see Figure 1 and Figure 2). The contribution weight for the episodic memory knowledge digraph is 1 and the contribution weight for the causal knowledge digraph is 2 . These two knowledge digraphs are used to de ne the connectivity pattern between the working memory nodes and the evidence nodes in Figure 3. In particular, the "total connectivity" pattern between the working memory nodes and the evidence nodes is expressed as a superposition (i.e., weighted sum) of the connectivity pattern de ned by the episodic memory knowledge digraph and the connectivity pattern de ned by the causal knowledge digraph. The weights in this weighted sum are the episodic memory knowledge digraph contribution weight 1 and the causal knowledge digraph contribution weight 2 . It is important to emphasize that the connectionist network in Figure 3 is not a fully connected network where every possible connection weight is "learned" (i.e., estimated). Rather, there are only two parameters ( 1 and 2 ) in the network in Figure 3 which completely specify (in conjunction with the episodic memory and causal knowledge digraphs) the entire connectivity pattern of the network. There are two important advantages associated with the dramatic reduction in the number of free parameters. First, since every free parameter of the KDC analysis is interpretable as the contribution of a speci c knowledge digraph, this means that the results of a KDC analysis are almost always interpretable in a straightforward way. Second, since a protocol corresponds to one data point in the KDC analysis, it is necessary to have at least as many data points as free parameters. Reducing the number of free parameters in the KDC model results in a dramatic improvement in the reliability of statistical inferences for a given sample set (i.e., given set of data points). On the other hand, a potential problem with using any statistical model with a small number of free parameters to explain a complex phenomenon (such as sequential Knowledge Digraph Contribution Analysis 14 structure in protocol data) is that the statistical model will not provide a good t to the data. In such a situation, using classical methods of statistical inference, reliable statistical inferences are not possible. For this reason, special methods of statistical inference which permit the testing of hypotheses in the presence of model misspeci cation are employed (see Golden, 1995, 1996 and White, 1982, 1994 for relevant literature reviews; also see the Appendix). Distributional assumption 4: Response competition assumption. It is important that the KDC analysis make predictions about speci c occurrence probabilities of propositions in a protocol. Such predictions are important for the purposes of parameter estimation since the contribution weights are selected to minimize the distance between the model's predicted probabilities and the database's observed relative frequencies. The activation levels of the evidence nodes in Figure 3 are not probabilities. The activation level of an evidence node may be greater than one, less than zero, and moreover the sum of the activation levels of all evidence nodes is not guaranteed to add up exactly equal to one. Thus, a transformation is required to map the activation levels of the evidence nodes into a new set of activation levels which are probabilities. The transformation must also have the property that an increase (or decrease) in the activation level of an evidence node must result in an increase (or decrease) in the activation level of that evidence node's respective probability node. Such a transformation is depicted in Figure 3 by the connections between the evidence nodes and the probability nodes. The connections from the evidence nodes to the probability nodes in Figure 3 are not modi able. Rather, the connectivity pattern is xed and may be visualized as a type of forward lateral inhibition connectivity pattern. Activating one evidence node in the network causes the corresponding probability node to become more activated while simultaneously decreasing the activations of the other probability nodes so that the sum of the activation levels of all probability nodes is always exactly equal to one. Psychologically, the forward lateral inhibition component of this distributional assumption is not inconsistent with experimental evidence that people make elaborative inferences in certain types of predictive contexts during text comprehension (McKoon & RatKnowledge Digraph Contribution Analysis 15 cli e, 1989; Singer & Ferreira, 1983; Van Den Broek, 1990). The speci c functional form of the nonlinear transformation from evidence node activation levels to probability node activation levels (see Appendix for additional detals) is similar to speci c probability distributional assumptions which have been used successfully in the Search of Associative Memory (SAM) model of free recall (Gillund & Shi rin, 1984; Mensink & Raaijmakers, 1988; Raaijmakers & Shi rin, 1981) and in the construction phase of the Construction Integration (CI) model (Kintsch, 1988). KDC Analysis Methodology Estimation of Contribution Weights A simple formula for computing the contribution weights for a given KDC model does not exist. On the other hand, relatively complex numerical optimization algorithms for estimating the contribution weights are available. Regardless of the speci c choice of numerical optimization algorithm, the same contribution weights will always be obtained under fairly general conditions. This is an important observation since this means that the speci c numerical values of the estimated contribution weights are meaningful. Unlike classical methods of statistical inference, the optimal contribution weights are de ned as those weights which make the observed data most likely under the KDC distributional assumptions. That is, the optimal contribution weights are quasi-maximum likelihood estimates (see Golden, 1995, 1996, for a review). These contribution weights can be shown to provide a "bestt" between the KDC predicted probabilities and the observed relative frequencies. Problems with estimating contribution weights. As alluded to earlier, there are two important concerns about estimating contribution weights using the KDC analysis. The rst concern is whether the speci c numerical optimization algorithm which has been chosen to estimate the contribution weights has successfully converged to the optimal weights. In order to verify successful convergence, it is necessary to show that small perturbations to the contribution weights have negligible e ects on the t of the model to the data according to the KDC goodness-oft measure. A typical test for Knowledge Digraph Contribution Analysis 16 establishing convergence is to evaluate the gradient norm of the KDC goodness-oft measure at the candidate contribution weights and show that such a norm is su ciently small. It can be shown that correctly designed numerical optimization algorithms will always successfully converge to the optimal contribution weights provided su cient computational resources are available. The second concern is whether the contribution weights are unique. Suppose a researcher proposes 30 knowledge digraphs corresponding to 30 contribution weights and proposes to estimate the values of these 30 contribution weights using 2 recall protocols. Although any well-designed numerical optimization algorithm will converge to a solution for the 30 contribution weights, it can be shown that such a solution will never be unique. Moreover, although having an equal number or fewer contribution weights than the number of protocols is a necessary condition for solution uniqueness it is still not a su cient condition. In order to test uniqueness of the contribution weights, one can see if any small perturbation of the contribution weights always results in a decrease in the t of the model to the data according to the KDC goodness-oft measure. A typical measure of this property is the condition number for a KDC model whose value ranges from 1 to in nity. Condition numbers which are less than 100 are usually viewed as associated with "well-conditioned" models where the contribution weights are unique. KDC models that are not well-conditioned are analogous to linear regression models which show evidence of multicollinearity. Estimating the contribution weights for a real database. In order to illustrate the use of the KDC analysis, the Golden et al. (1993) text recall data obtained from 24 University of Texas at Dallas college students was re-analyzed using KDC analysis. The four texts used in this study were the Fox and Bear text, the Epaminondas text, the Judy's Birthday text, and the Tiger's Whisker text. These texts were obtained from the Stein and Glenn (1979) study. Subjects read and recalled two of the four texts in an immediate recall condition. The subjects then returned 1-week later to recall the same two texts again in a delayed recall condition. Subjects were given a pen and paper to generate written recall protocols. Subjects were also instructed to write their recall of the text so that the order of events in the recall protocol accurately Knowledge Digraph Contribution Analysis 17 re ected the order of events in the original text. Causal network analyses of the four texts were generated using the methodology of Van Den Broek (1990; also see Trabasso, Secco, and Van Den Broek, 1984). Beta contribution weights for all four texts were successfully estimated and all four KDC models were well-conditioned. Figure 4 shows the contribution weights estimated for the episodic memory knowledge and causal knowledge digraphs for the four texts in two experimental conditions: Immediate recall and delayed recall. A number of researchers (see Van Den Broek, 1990, for a review) have found that as retention interval increases, that the percentage of statements which have fewer causal connections with other statements in the text are less likely to be included in a subject's recall protocol. Kintsch & Van Dijk (1978), as well as other researchers, have shown that propositions pertaining to the main ideas in a text tend to be remembered more e ectively than propositions which are not as directly relevant to the main ideas in the text. Figure 4 shows the results of the knowledge digraph analysis in this particular situation. A large contribution weight for the causal knowledge digraph indicates that the order of the propositions generated by the subject is very consistent with the causal structure of the text. Thus, the causal knowledge digraph weight may be interpreted as an in uence of the "macrostructure" of the text. A large contribution weight for the episodic memory knowledge digraph indicates that subjects merely "memorized" the text and recalled the text in exactly the same order in which it had been presented to them. As shown in Figure 4,in almost all cases, the contribution of the episodic memory knowledge digraph tends to decrease which implies that subjects forget the original ordering of propositions in the text as retention interval increases. In addition, the contribution of the causal knowledge digraph either remains constant or tends to increase implying that causal knowledge structural relationships among propositions in the text continue to strongly in uence how subjects recall the text as retention interval increases. This pattern of results is consistent with the nding that as retention interval increases, details of the text which are unimportant to the plot of the text tend to be forgotten. Knowledge Digraph Contribution Analysis 18 ||||||||| Insert Figure 4 About Here ||||||||| Selecting the "besttting" KDC Model A statistical test for model selection. A KDC probability model is de ned by: (i) a proposition table, (ii) a set of knowledge digraphs, and (iii) a set of working memory local coherence assumptions. Suppose we are given a set of protocol data and two KDC probability models. We would like to do a statistical test to decide which of the two KDC probability models "bestts" the unobservable protocol data generating process by examining a data sample of N protocols. Such a statistical test would have three possible outcomes: (i) the rst KDC model accounts for the data better than the second KDC model, (ii) the second KDC model accounts for the data better than the rst KDC model, and (iii) both KDC models account for the data equally well. Notice that in the latter case where both KDC models account for the data equally there are two possible interpretations. It may indeed be the case that both KDC models account for the data equally well but it is also possible that due to lack of statistical power one model actually does account for the data better than the other model. We would also like a statistical test which does not require the two KDC models to be "nested" (i.e., that the set of knowledge digraphs for one KDC model is a subset of the set of knowledge digraphs for the other KDC model). We also would prefer a statistical test which does not require that at least one of the two KDC models provides a "goodt" to the data. Such a large-sample statistical test for simultaneously addressing both of these issues has been developed using Vuong's (1989) asymptotic statistical model selection theory (see the Appendix for additional details). A speci c recency local coherence strategy. All results reported in this paper were based upon a very simple recency local coherence strategy which is now de ned in detail. Consider a protocol which consists of a sequence of propositions, followed by some particular proposition called P , and then followed by another sequence of DIST Knowledge Digraph Contribution Analysis 19 propositions, and then a target proposition. Assume proposition P occurs exactly once in the protocol. One particular local coherence recency strategy theory about the activation level of the working memory node corresponding to proposition P at the time when the target proposition is being predicted might be implemented by assuming proposition P 's working memory node has an activation level of DIST . The constant , whose value ranges between zero and one, is called the working memory decay rate. Thus, if = 1 activation levels of nodes in working memory do not decay at all. If = 0, then the activation level of working memory node P initially activated in working memory decays so rapidly that the activation level of working memory node P is equal to zero if the target proposition is separated from proposition P in the protocol by one or more propositions. The model selection procedure for selecting decay rates. Di erent working memory decay rates correspond to di erent KDC models. In the models considered here, ve possible values of were considered = 0, = 0:1, = 0:2, = 0:3, = 0:4, and = 0:5. The working memory decay rate with the smallest value of the KDC goodness-oft error measure was then selected for each of the four texts. The optimal working memory decay rate for the Epaminondas, Tiger's Whisker, and Fox and Bear texts was found to be = 0:3. The optimal working memory decay rate for the Judy's Birthday text was = 0:2. It is interesting to note that the working memory decay rate was approximately the same value for the four texts in this study. Most likely, working memory decay rate will vary as a function of the text and the experimental conditions under which the text is read. However, further research into the issue of when model parameters such as working memory decay rate are invariant across texts and speci c experimental conditions would be highly desirable. In this initial study, the working memory decay rate was about = 0:3 which means the activation level of the previously recalled statement is equal to 30% of its original activation level. This suggests that only the last 1:3 statements which were most recently mentioned are maintained in working memory during recall. This pattern of results seems qualitatively similar to results reported by Glanzer, Knowledge Digraph Contribution Analysis 20 Dorfman, and Kaplan (1981). Glanzer et al. (1981) periodically interrupted subjects while they listened to texts and had subjects recall one of the last four sentences they heard. Glanzer et al. (1981) found subjects could accurately recall about two simple sentences. At this stage, however, more systematic studies which estimate the working memory decay rate in a variety of situations for a variety of texts are required before one interprets the working memory decay rate as an actual task-dependent and textdependent measure of human working memory capacity. Next, for each text, the KDC error measure associated with the optimal working memory decay rate was subtracted from the KDC error measure computed for the ve other values of the working memory decay rate. The resulting ve numbers are normalized KDC errors. It can be shown that a normalized KDC error is formally equivalent to a particular type of likelihood ratio which compares the likelihood of the optimal model with respect to each of the suboptimal models. Figure 5 shows histograms of the normalized KDC error measure for each of the suboptimal models as a function of text and working memory decay rate with 68% (i.e., plus or minus one standard error) con dence intervals. Examining both the values of the normalized KDC error measure and the con dence intervals for each of the suboptimal models shown in Figure 5, one notes that the di erences between the = 0:2 and = 0:3 conditions were comparable for the Judy's Birthday, Tiger's Whisker, and Fox and Bear texts. Figure 5 also reveals an advantage for activating more than just the most recently active proposition in working memory to predict the next subsequent proposition (compare the = 0 case with the = 0:2 and = 0:3 cases). Finally, note that although only con dence intervals are presented in this paper, KDC analysis also provides mechanisms for testing explicit hypotheses regarding which of two KDC model's "bestts" a given data set. It should also be emphasized that the same techniques which were used in this section can be used to decide which of several knowledge digraph representations "bestts" a given data set. This approach was used by Golden (1994) to compare a su ciency-type causal network analysis as described by Graesser (1978) with the counterfactual-type causal network analysis described by Knowledge Digraph Contribution Analysis 21 Trabasso, van den Broek, and their colleagues (see van den Broek, 1990, for a relevant review). ||||||||| Insert Figure 5 About Here ||||||||| Testing Hypotheses about the Contribution Weights The contribution weights reported in Figure 4 were obtained by estimating the contribution weights for each of six di erent values of the working memory decay rate and then using those contribution weights associated with the besttting value. Although Figure 4 is informative, Figure 4 does not provide insights into which of the observed di erences are real and which di erences are due to chance (i.e., sampling error). In order to address this type of issue, standard errors for the parameter estimates were mathematically derived using White's (1982) asymptotic statistical theory in conjunction with the KDC distributional assumptions. Wald tests were then used in conjunction with the parameter estimates, standard errors, and estimated covariance matrices to derive appropriate statistical tests. A computer program was then developed to facilitate the use of these mathematical formulas in the analysis of protocol data. Please see the Appendix, Golden (1994, 1995, 1996), and White (1982, 1994) for additional details of the mathematical formulas implemented in the computer program. Planned comparison: E ects of retention interval. The KDC analysis has mechanisms for constructing a statistical test involving several contribution weights which does not require averaging contribution weights. Statistical tests for comparing how one set of weights changes as a function of retention interval can be constructed. A planned comparison test investigated if the set of causal knowledge digraph contribution weights and episodic memory knowledge digraph contribution weights varied as a function of retention interval. This statistical test was only marginally signi cant Knowledge Digraph Contribution Analysis 22 (W (8) = 15:0; p = 0:06). The other statistical tests reported in the following sections were post-hoc comparisons designed to further investigate and understand this marginally signi cant di erence. Individual contribution weights. There were four episodic memory knowledge digraph contribution weights in the immediate and delayed recall conditions corresponding to each of the four texts. Similarly, there were four causal knowledge digraph contribution weights estimated in the immediate and delayed recall conditions as well. Of the 16 contribution weights, 15 of the contribution weights were signi cantly different from zero at the 0.05 signi cance level. The only digraph contribution weight which was not signi cantly di erent from zero at the 0.05 signi cance level was the causal knowledge digraph contribution weight estimated for the Fox and Bear text in the immediate recall condition (W (1) = 0:65; p = 0:4). These results are consistent with the hypothesis that both the episodic memory knowledge digraph and the causal knowledge digraph contained information which explained sequential regularities in the observed protocol data for almost all texts and experimental conditions. These results are consistent with previous research (see Van Den Broek 1990, for a relevant review) which has demonstrated that a causal network analysis of a text can account for some of the properties of a text which make that text memorable. Other across texts analyses using sets of contribution weights. The set of four episodic memory knowledge digraph weights tended to decrease in value as retention interval increased (W (4) = 9:3; p = 0:05), while the set of causal knowledge digraph weights increased in value as retention interval increased (W (4) = 11:2; p = 0:03). Across texts analysis using averaged contribution weights. The values of knowledge digraph weights averaged across the four texts were then computed. In the immediate recall condition, the average episodic memory knowledge digraph weight (M = 3:5) was signi cantly di erent from zero (W (1) = 954; p < 0:001) and the average causal knowledge digraph (M = 0:60) weight was signi cantly di erent from zero (W (1) = 27:7; p < 0:001). In the delayed recall condition, the average episodic memory knowledge digraph weight (M = 3:1) was signi cantly di erent from zero (W (1) = 798; p < 0:001) and the average causal knowledge digraph weight (M = 1:0) Knowledge Digraph Contribution Analysis 23 was signi cantly di erent from zero (W (1) = 95; p < 0:001). In addition, the average episodic memory knowledge digraph weight decreased as retention interval increased (W (1) = 8:1; p < 0:005), while the average causal knowledge digraph weight increased as retention interval increased (W (1) = 7:3; p < 0:01). Analysis of individual texts. Examination of individual texts only showed a signi cant e ect (at the 0.05 level of signi cance) of individual contribution weights as a function of retention interval for the Judy's Birthday text. The episodic memory knowledge digraph weight for the Judy's Birthday text decreased in value as retention interval increased (W (1) = 3:7; p = 0:05), while the causal knowledge digraph weight increased in value as retention interval increased (W (1) = 8:4; p < 0:005). In order to examine the extent to which the Judy's Birthday text was contributing to the marginally signi cant planned comparison, two additional post-hoc comparisons were done which omitted the Judy's Birthday text in the analyses. Without the Judy's Birthday text, the average episodic memory knowledge digraph weight decreased as retention interval increased (W (1) = 5:0; p = 0:03). On the other hand, omission of the Judy's Birthday text in the computation of the average causal knowledge digraph contribution weight showed that the average causal knowledge digraph weight increase as retention interval increased was only marginally signi cant (W (1) = 2:4; p = 0:1). Discussion. The purpose of these statistical analyses was to illustrate the application of KDC analysis for the analysis of protocol data. Consistent with ndings in the literature, the relative contribution of the causal knowledge digraph contribution weight was positive and increased in value as retention interval increased. The planned comparison was only marginally signi cant. The lack of statistical power in the analysis is most likely due to a combination of factors which include: (i) the quality of recall data, (ii) the number of protocols per individual text, (iii) the quality of the knowledge digraph models, and (iv) the fact that most of the texts were very causally coherent. This latter point requires some further discussion. Presumably, less causally coherent texts would show greater changes in contribution weights as a function of retention interval since, as retention interval increases, subjects would have a greater tendency Knowledge Digraph Contribution Analysis 24 to restructure their protocols in order to t their underlying causal schemata (e.g., Bischofshausen, 1985; Bower, Black, & Turner, 1978; Levorato, 1991; Lichtenstein & Brewer, 1980; Mandler, 1978; Mandler & DeForest, 1979). Some support for this hypothesis comes from examination of the texts in this study. The Epaminondas text was very causally coherent and for this text both episodic memory and causal contribution weights did not seem to change as a function of retention interval (see Figure 4). On the other hand, the Judy's Birthday text was less causally coherent and the e ects of retention interval on the episodic memory and causal knowledge contribution weights for this text were quite pronounced. Psychological and Computational Validation of KDC models Issues of model reliablity were addressed in the previous sections. This section is concerned with the issue of model validity. Model validity is a subjective quantity which will vary (for complex models) as a function of the goals of the researcher. In this case, obtaining a consistent measurement of a knowledge digraph weight is an example of model reliability. Demonstrating that a knowledge digraph weight is related in a meaningful way to the performance of relevant and interesting cognitive tasks is an example of model validity. The procedure for validating a KDC model is based upon a two step procedure. First, generate "synthetic" recall protocols from a KDC model. Second, compute statistics using the synthesized recall protocols and compare those statistics to similar statistics computed using a set of human recall protocol data. Step 1: Generate synthetic protocols. The KDC model in conjunction with a set of speci c contribution weights de nes an explicit probability distribution which implicitly assigns a probability mass (i.e., a "likelihood") to a given protocol (i.e., ordered sequence of propositions). This probability distribution can be sampled to generate synthetic protocols. In particular, the probability of every possible ordered sequence of propositions which could be generated is estimated and each ordered sequence of propositions is assigned to the side of a very high-dimensional die. For example, for ordered sequences of no more than 10 propositions and for a proposition table with no Knowledge Digraph Contribution Analysis 25 more than 15 propositions, the number of sides of the die would exceed 15 10 (which is a very large number). The die is rolled, and the outcome of the die roll is recorded as a synthesized protocol. Relevant statistics (e.g., statement recall probabilities) computed from the synthesized protocol data may then be compared with human subject generated protocol data. Moreover, the synthesized protocols may be inspected from the perspective of an engineer interested in designing arti cially intelligent systems that can recall, summarize, and answer questions about texts. The validity of the knowledge digraphs in conjunction with the KDC model can thus be evaluated from a computational perspective. The method for generating synthetic protocols described above, although useful for expository reasons, is computationally intractable. Several computationally tractable methods for sampling protocols from the KDC model are currently under development. The speci c algorithm which was used in the simulation results reported here may be viewed as a stochastic version of the connectionist relaxation algorithm described by Golden and Rumelhart (1993) and Golden (1997) for generating recall protocols (see Appendix for additional references). Step 2: Validate knowledge digraph using synthetic protocols. In order to illustrate how the KDC sampler algorithm may be used for KDC model validation, a "synthesized" database was generated from the KDC sampler algorithm for the Fox and Bear text. The synthesized database consisted of 48 recall protocols (i.e., 48 ordered sequences of text propositions) such that half of the recall protocols were generated from the human immediate recall data, and the remaining half of the recall protocols were generated from the human delayed recall data. Thus, each "synthesized" recall protocol generated by the KDC sampler algorithm and the knowledge digraphs could be yoked to a corresponding human recall protocol. Both the human and KDC model recall data were analyzed in exactly the same manner. The 10 propositions in the Fox and Bear text which had three or more causal connections with other propositions in the text were de ned as high-connectivity propositions, while the remaining 15 propositions were de ned as low-connectivity propositions. The percentage of times a statement was recalled was then computed for both Knowledge Digraph Contribution Analysis 26 the immediate recall condition and the 1-week delayed recall condition. The data was then analyzed using an unbalanced two-factor (causal connectivity by retention interval) within-subjects ANOVA for both the human recall data and the model recall data. A summary of human and model recall performance as a function of causal connectivity and retention interval is shown in Figure 6. |||||||||||| Insert Figure 6 About Here |||||||||||| Humans recalled more high-connectivity propositions (M = 0:67) than low-connectivity propositions (M = 0:55), F (1; 23) = 17:97; p < 0:01. In addition, humans recalled more propositions in the immediate recall condition (M = 0:62) relative to the delayed recall condition (M = 0:58), F (1; 23) = 14:21; p < 0:01. The interaction between causal connectivity and recall condition was not signi cant, F (1; 23) = 0:96; p = 0:37. The recall data generated by the KDC sampling algorithm with respect to the digraphs described in Figures 1 and 2 was then analyzed. The model recalled more highconnectivity propositions (M = 0:53) than low-connectivity propositions (M = 0:41), F (1; 23) = 14:22; p < 0:01. However, the number of recalled propositions did not vary as a function of retention interval, F (1; 23) = 1:23; p = 0:28. The interaction between causal connectivity and recall condition was not signi cant, F (1; 23) = 0:22; p = 0:64. Thus, the model demonstrated an e ect of causal connectivity but not of retention interval. Most likely this is due to lack of statistical power since only one text was considered in this simulation study. It should also be noted that the causal knowledge digraph contribution weight did not signi cantly change as a function of retention interval.