Ubiquitous Complex Event Processing in Exocortex Applications and Mathematical Approaches

The concept of an artificial cognitive system uCepCortex is discussed that integrates multiple sources of information, including from specialized sensors and software agents in the cloud, and acts as an exocortex for human users. It combines a high degree of autonomy with sophisticated human interface techniques to ensure that the human is kept in the loop and exercises control, while greatly extending his/her effective cognitive range by off-loading processes to the cognitive system. The uCepCortex approach utilizes the emerging disciplines of Ubiquitous Complex Event Processing and Event-Driven Process Management to implement a human-inspired cognitive model that complements human abilities and can detect, correlate, filter, enrich, process and learn from millions of events per second from arbitrary event types, and that implements flexible (re-) actions or processes in rapidly changing scenarios. Finally we propose a mathematical model for a neuro-bio-ICT system, as an application of the theory of Memory Evolutive Systems and its handling of uCepCortex applications. KeywordsExocortex, artificial cognitive system, complex event processing, FPGA, belt, cloud-based, agents, ubiquitous, smart sensors, brain computer interface, fuzzy cognitive maps, Memory Evolutive Systems, mathematical modelling I. “ARTIFICIAL COGNITIVE SYSTEM” (ACS) A brainstorming on the definition of ACS was started in connection with the ICT Workprogramme of the European FP7, Call 9, Challenge 2.1 "Cognitive Systems and Robotics" [1]. Objective 2.1b specifically focusses on ACS. That is what we address with the uCepCortex approach. As there is no generally accepted definition of ACS, we define in the following an ACS as a system which (i) is artificially made by humans (ii) is designed to enhance the cognition or abilities of humans (iii) has one or more explicitly defined aims (iv) can recognize or even predict or foresee complex events (e.g. environmental situations, natural catastrophes, economical or political forecasts, health hazards) and (v) can react to such complex events as a self-contained autonomous system, or can notify affected humans and provide helpful advice to aid decision-making. II. MISSION OF THE UCEPCORTEX APPROACH The main idea of the uCepCortex project is to enhance human abilities by a complex cognitive system which adds and supplements senses and integrates the information of these senses for an optimal output, thereby overcoming the limitations of the human brain. Its typical limitations are: • the amount of events that can be processed at any one time: The human brain can only process around 120.000 events per second unconsciously and less than ten consciously; • the number of event types we can consciously integrate at any one time: integrating of multiple information sources requires tracking, memorizing and retrieving of past, recent and current events; • the performance and scalability of the event processing and the correlation of apparently meaningless basic events to senseful complex events; • the sensitivity ranges (modalities) of the five senses what a human can hear, see, smell, taste and feel; • the degeneration of the number and the sensitivity of senses: loss or damage due to aging, illness or accident; the inability to substitute or to add additional senses. The evolution of the human brain and the human senses are responsible for how humans construct their reality and how they react to events – sensibly, or counterproductively, or even resulting in mental diseases. The “mechanisms” of the so far evolved brain seem to be well adapted to filter hundreds of thousands of events per second and to react only on those we must process consciously according to a specific situation, or automatically and unconsciously when our executive brain would work too slow – e.g. in the case of a danger or emergency or when a complex event pattern has already been processed in the past and must not be consciously processed again. These mechanisms have worked more or less efficiently and effectively for around 40.000 years. However, the accelerating complexity of human life, society, economics, communication and ready access to virtualized information requires the enhancement of human mental abilities, and of the much too slow cognitive-physiological evolutionary process. Therefore a cognitive system should not mimic the human brain in such a context, but should provide the missing components and capabilities as illustrated in [2]. The uCepCortex project develops an artificial cognitive system that integrates multiple sources of information, including from specialized sensors and agents in the cloud, providing an exocortex for humans. It combines a high degree of autonomy with sophisticated human interface techniques to ensure that the human is kept “in the loop” and exercises control, while greatly extending the effective cognitive range by off-loading processes to the cognitive system. The uCepCortex approach utilizes the emerging disciplines of Ubiquitous Complex Event Processing and Event-Driven Process Management to implement a cognitive model that complements human abilities and can detect, correlate, filter, enrich, process and learn from millions of events per second from arbitrary event types, and that implements flexible (re-) actions or processes in rapidly changing scenarios. It includes the development of low-cost smart sensors that may be either worn or distributed in the environment, including some extending senses beyond human abilities, and that implement cognitive processes including attention, collision avoidance, recognition, anomaly detection and detection of emotional cues, thus providing high-value information to the human users. These components of the exocortex use rapid, parallel and reconfigurable computational hardware (FPGA) to implement the artificial cognitive architecture. The uCepCortex cloud-based agents provide services for interaction with wider information systems, including interaction with other uCepCortex agents and social media, integrating this with sensory data. The exocortex includes rich information flow to the human, and limited control flow from the human, consistent with emerging interface technology. System to human interfacing includes rapid, high-bandwidth augmented sensory input via a belt device, and linguistic/visual feedback for symbolic information. Human to system control will include audio command and traditional input devices. The uCepCortex cognitive system is designed to complement human cognitive processes. We will investigate how the brain responds to exocortex interaction using emerging Mind Reading technologies as realized in brain-neural computer interfacing approaches (see website about future BNCI), and uCepCortex learning algorithms will support coadaptation of the exocortex and user. uCepCortex capabilities are generic and can ultimately be used to address a wide range of needs. We include two types of demonstrators exploring improved quality of life: Impaired subjects: For subjects with restricted sensory or cognitive ability, uCepCortex will provide compensatory capabilities including presence of other humans, recognition, social context and interpretation of non-verbal emotional cues. Emergency management: For subjects whose roles present high cognitive demand, uCepCortex will integrate a broad range of sensory and information inputs allowing real-time exploration and control of a complex, changing environment. Special work packages of the project, which cannot be covered in this paper and are subjects of dedicated submissions to specific conferences, deal with the definitions of • Deterministic [3] and non-deterministic reference models of uCepCortex applications [4] • a reference architecture based on an enhancement of the NEXOF-RA [5] • a standard Notification Event Architecture of Thought (NEAT) as a class diagram following the idea of NEAR [5] • a U-CEP modeling notation as a suggestion of a future OMG standard [4]. III. MULTI-LEVEL DYNAMIC COGNITIVE MODELING Mathematical modeling of cognitive processes is still an open challenge towards the long-term vision of understanding, simulating and interacting with the human brain. Studies in cognitive psychology suggest that a cognitive model should be considered at multiple levels [6], be aware of uncertainty [7], and allow for adaptation to changes in the surrounding environment [8]. In this context the lowest level of cognitive processing deals with multi-source sensory information from the environment, and propagates that information to the higher cognitive processing levels after abstraction. The higher cognitive processing levels enable representation of, and reasoning, between semantically expressible concepts, such as objects, properties, relations, actions and conditions. The uCepCortex concept accepts the challenge to investigate multilevel cognitive modeling through a novel mathematical formalism that enables complex event processing in dynamically changing environments. System ubiquity, adaptability, autonomy, and interactivity between system components and between the system and the environment can be facilitated by a hardware/software multi-agent implementation featuring cloud/grid connectivity, efficient resource allocation and high throughput event processing capabilities [9]. A. Low-level cognitive modeling Low level cognitive modeling should consider mathematical representations of events from multiple sources/modalities, including sensors of visual and physiological signals. Signal filtering, feature extraction and feature fusion techniques, are considered as means to develop such representations through efficient machine learning approaches enabling identification of complex patterns, knowledge extraction and system personalization for different users. The uCepCortex concept includes a distributed cognitive system consisting of two types of components: smart sensors and cloud-based agents, integrated in a CEP architecture. To apply the CEP model, sensor input streams must first be processed to identify low-level (atomic) events. We identify an appropriate set of atomic events that: a) are semantically meaningful, conveying significant information in isolation or combination; b) can be derived efficiently from sensor input streams. The set of events include a variety of attributes, such as visual attributes; motion signals; detection, tracking and identification of objects of interest, e.g. humans, facial indicators correlated with their emotional state and objects in their environment that would enable inference about their activities or the context of their activities. In order to cope with the challenge of adaptation to real-world changing environments, noise tolerant, uncertainty-aware, feature representations that are invariant to environmental changes will be considered [10, 11, 12]. Further challenges arise considering the multitude of the available data sources and the respective feature representations, demanding effective and informative feature fusion methods that enable discriminative cross-feature space representations for machine learning [13, 14, 15]. The low-level event dictionary includes a mixture of generic, lowlevel events that may be of widespread use; and more specialized events servicing particularly high-value needs e.g. detection of non-verbal cues from human interlocutors. The resulting event stream can be further processed by the uCepCortex distributed cognitive architecture and/or conveyed to the human user via a brain-computer interface. The sensor analysis includes attentional mechanisms, allowing a focus on salient parts of the signals, driven both bottom-up (i.e. by anomaly or significant pattern identification) and top-down (by attentional prompts from the user). Challenges regarding the machine learning methods to be investigated include: a) identification of complex event patterns corresponding to concepts, necessary for inference by the high-level cognitive models to be developed; b) extraction of knowledge about cognitive processes and the related concepts, enabling construction of high-level cognitive models; c) knowledge transfer between subjects and personalization of the system to be developed according to subject-specific characteristics. The efficiency of the machine learning methods in the context of low-level cognitive processing can be investigated from both an algorithmic and an implementation perspective. B. Higher-level cognitive modeling Higher-level cognitive tasks include reasoning and knowledge inference for decision making and complex system control in real-world conditions. Challenges include automatic or semi-automatic construction of cognitive models, selfadaptation, integration of low-level cognitive processing methods and evolutionary optimization processes. Towards this direction we consider uncertainty modeling as a priority to develop robust approaches being able to cope with indeterminacy, imprecision and missing information. The theory of fuzzy sets provides a sound mathematical framework for uncertainty modeling that has proved its effectiveness in a variety of applications. Fuzzy knowledgebased reasoning methods require that knowledge is represented in the form of rules between higher-level concepts [16]. Machine learning-based methods are considered for the development of hybrid approaches that enable dynamic adaptation of the fuzzy knowledge models to changing environments [17]. The fuzzy cognitive map (FCM) approach can be the basis for enhanced networks for dynamic knowledge representation [18]. An FCM is a fuzzy directed graph with causally interrelated nodes that correspond to the concepts involved in a knowledge domain. It is able to reason through an iterative algorithm updating the values of the graph nodes until a steady state is reached [19]. Since the introduction of the original FCM model, several extensions have been proposed. In the context of uCepCortex, dynamic cognitive networks [18] could enable the definition of dynamic causal relationships, and a temporal concept can be exemplified by the timed automata-based FCMs [20] and a recent study on the temporal granularity of FCMs [21]. In [22] genetic algorithms have been considered for edge weight recalculation of the FCM, whereas the synergy of fuzzy modeling and evolutionary optimization has been highlighted for efficient design of FCMs in [23]. Recently, the mathematical framework of intuitionistic fuzzy sets (IFSs) has been applied by members of our consortium for modeling uncertainty in the context of FCMs [24, 25]. An IFS [26] is a generalized fuzzy set whose elements are characterized by both a membership and a non-membership degree to that set. The non-membership may not necessarily be symmetric to the membership, whereas a formal definition of hesitancy is obtained as a function of both of these quantifiers. It has been shown that this modeling approach enables a better approximation of human thinking by considering hesitation as part of intuitionistic reasoning and uncertainty propagation through the steps of the reasoning process [25]. The uCepCortex concept involves novel mathematical approaches for uncertainty modeling based on generalized fuzzy sets, such as IFSs, in the context of hybrid, dynamic and adaptive systems that combine both knowledge-based and machine learning methods. The proposed approaches are generic, extending well beyond uCepCortex, to a variety of intelligent applications. IV. MATHEMATICAL MODELING APPROACH The uCepCortex approach contains also a mathematical model of complex multi-scale, multi-agent systems with UCEP, such as biological, cognitive or social systems, in view of enhancing their capabilities, performance and control. Traditional mathematical models are well adapted at a local level, but an integrative approach is required in multi-degreeof-freedom systems with interactions backfiring between several levels and temporalities. The objective is to develop new models, based on recent mathematical domains (e.g., category theory), to understand the organizing principles of high level cognitive systems and how U-CEP affects their behavior, while suggesting ways to counteract the deficiencies and implement higher capabilities in an engineering sense. The Memory Evolutive Systems (MES) give such a model for multi-scale systems with a tangled hierarchy of components varying over time, self-organized by a network of internal agents with different rhythms, functions and logics, in which U-CEP plays an important role [28, 29]. A particular application of MES is the model MENS for a neuro-cognitive system. It singles out two characteristics of the human brain which have been proved essential for the emergence of high level cognitive processes and U-CEP: (i) Degeneracy: this is a kind of "flexible redundancy", defined as: "the ability of elements that are structurally different to perform the same function or yield the same output, [...] a ubiquitous biological property, [...] it is a feature of complexity both necessary for, and an inevitable outcome of, natural selection" [30]. This property ensures robustness and flexibility, and allows for the emergence over time of cognitive processes of increasing complexity [28]. (ii) Existence of a Central Core with several hubs forming a "rich club", discovered in 2008: "existence of a structural core in human cerebral cortex [...] both spatially and topologically central", "linked to self-referential processing and consciousness." [31]. This allows for the development of an 'internal model' integrating knowledge of different modalities, and is at the root of consciousness and anticipation [32]. Similar characteristics must be imposed when modeling high level cognitive systems, in particular neuro-bio-ICT systems. This can be explained using a general MES. The configuration of a MES at a time t is a category (= graph with a composition of successive arrows satisfying some associativity and identity axioms) [33]: its objects represent the states of the components at t, and the arrows (or links) are channels through which they can communicate. A link has a propagation delay and a force, and it is active or passive at t. The components are distributed in a finite number of levels so that the components of a given level are homogeneous between them, but more 'complex' than those of lower levels. Formally, a component C of level n+1 acts as the aggregate (or categorical "colimit" [33]) of a pattern P of linked lower level components, so that P, operating as a whole, and C, by itself, have the same functional role. The degeneracy property is translated into the following Multiplicity Principle (MP): there are multiform components C which admit several decompositions into structurally different and non well-interconnected patterns of lower levels, and C can operate through one or the other, and even switch between them depending on the context. MP gives flexibility and robustness to the system, and it is necessary for the emergence of components of higher complexity order (cf. [28]). The change of configuration from t to t' consists in events of the following kinds: addition, suppression or decomposition of some components, formation of complex components by binding or strengthening some pre-existing patterns. It is modeled by the complexification process [28] which explicitly describes the new configuration of the system after realization of a procedure with objectives of this sort. It is computable, for instance using topologically inspired languages such as MGS. The dynamic is modulated by the cooperation/competition between a net of internal co-regulators (CR); they operate with the help of a central dynamic memory, a sub-system whose multiform components represent knowledge of different modalities in a robust and flexible manner (thanks to MP). A CR is a functional sub-system which acts stepwise at its own rhythm. A step from t to t' consists of different phases: (i) Reception and analysis of the partial incoming information transmitted to CR through active links, leading to the formation of the landscape of CR at t (modeled by a category). (ii) Selection of an admissible procedure Pr with the help of the memory; the commands of Pr are sent to effectors. (iii) At the beginning t' of the next step, evaluation of the result. In the event the result is not adequate, there is a fracture for CR. One cause of fracture is the fact that each of the several coregulators operates with its own rhythm and local logic. As all their commands at a given time can be conflicting, an interplay among the co-regulators is necessary, leading to a kind of Darwinian selection among their commands, made flexible enough by the possibility of switches between different decompositions of the commands to select the most adapted ones. The interplay may pass over some commands, causing a fracture for the corresponding co-regulators. An important cause of fractures is non-respect of the synchronicity laws which relate the rhythm of a co-regulator to the propagation delays of the links and the stability spans of the components in its landscape. Fractures not repaired soon give de-synchronies, and can backfire through levels, forcing a cascade of re-synchronisations to co-regulators of increasing levels, a process at the root of a Theory of Aging [28]. Over time, a MES develops a central core, called the Archetypal Core (AC), through the emergence of components of higher complexity order integrating knowledge of different modalities; these components become connected by strong links forming loops which self-preserve their activation for a long time. AC plays the role of an 'internal model' reflecting the identity of the system; it plays a central role in the dynamics, taking part of the diffusion and processing of information. Indeed, an unexpected event will activate part of AC; this activation diffuses through self-maintained archetypal loops, and then propagates to lower levels through decompositions; it leads to the formation of a global landscape GL in which U-CEP takes place, through two intermingled processes: (i) a retrospection process for "sensemaking" of the event and its possible causes; (ii) a prospection process for developing adequate strategies to deal with it. The construction of the model MENS for a neuro-cognitive system relies on a common process in brain dynamics: a cognitive event depends on the activation of more or less complex and distributed neuronal assemblies acting synchronously (Hebb). The level 0 of MENS models the neural system Neur (whose components are the neurons and the links the synaptic paths between them). The higher level components are conceptual objects, called category–neurons, which code information under the form of more or less distributed neuronal (hyper-) assemblies acting synchronously. MENS is obtained by successive complexifications of Neur, and its Archetypal Core AC is based on the structural core of the brain (i.e., its components have ramifications down to it). A Neuro-Bio-ICT system is obtained by connecting the brain to an uCepCortex. Its functioning will be modeled by a MES, called NBIS, constructed by the same process that leads from Neur to MENS: we construct a large system ENeur, containing Neur and the uCepCortex, as well as the links between them; NBIS will be obtained by successive complexifications of ENeur. The preceding results show that, for a good handling of U-CEP by NBIS, both the uCepCortex and the whole system ENeur must satisfy the above Degeneracy and Central Core properties. This imposes sufficiently strict conditions on them and on the links between the uCepCortex and Neur. The results obtained for MENS (in particular the role of AC in the construction of a global landscape where to handle U-CEP) can be transposed to NBIS. NBIS might allow the development of new methods to increase mental and learning capacities, and to cope with mental deficits and impairments due to neuro-degenerative pathologies, such as Alzheimer or Parkinson. In aging it could help monitoring dysfunctions, e.g. those leading to resynchronizations, and allow for preventive treatment. V. CONCLUSIONS AND FUTURE WORK Mathematical approaches are essential when we must describe use cases on a higher and generalized abstraction level, which is the basis for a concrete realisation of applications in specific domains such as the bio-medical or socio-processes of our mentioned demonstrators. In this paper we concentrated on formal modelling aspects, focusing on the mathematical modelling approaches. Another formalism of the uCepCortex project is needed as a precondition to transform the models into an executable model. Future work will exemplify these mathematical approaches within our demonstrators, e.g. emergency management. Such applications could never work without such a basic foundation; they would be "hand-made" and tuned for a specific situation, or for a specifically anticipated range of circumstances, which is not a sufficiently adaptive approach to deal with the complexities of real world events.