BIDCEP: A Vision of Big Data Complex Event Processing for Near Real Time Data Streaming

This position paper aims to trigger a technical discussion by proposing a conceptual architecture for big data streaming integrated with complex event processing (BiDCEP). BiDCEP expands the Lambda and Kappa (LK) architectures for big data streaming to fit the complex event processing (CEP) and event management domains of enterprise IT. BiDCEP links CEP components as defined in previous work of Events Collections, Purifications and Enrichments with the big data LK batch and speed layers, and wraps the LK service layer with integration interfaces for expandable grid of interlinked BiDCEP units. The BiDCEP architecture can enable the LK big data quality attributes of scale, availability and latency to be maintained, while accounting for CEP enterprise IT requirements of load and content shedding, basic and derived enrichment, semantics transformation, and security enforcement. As such, open source big data streaming strengths can be employed within the context of an enterprisegrade IT with monitored service levels.