Scalable In-Situ Exploration over Raw Data
نویسنده
چکیده
Application. The Palomar Transient Factory (PTF) project aims to identify and automatically classify transient astrophysical objects such as variable stars and supernovae in real-time. A list of candidates is extracted from the images taken by the telescope during a night. They are stored as a table in one or more FITS files. The initial stage in the identification process is to execute a series of aggregate queries over the batch of extracted candidates. This corresponds to data exploration. The general SQL form of the queries is:
منابع مشابه
OLA-RAW: Scalable Exploration over Raw Data
In-situ processing has been proposed as a novel data exploration solution in many domains generating massive amounts of raw data, e.g., astronomy, since it provides immediate SQL querying over raw files. The performance of in-situ processing across a query workload is, however, limited by the speed of full scan, tokenizing, and parsing of the entire data. Online aggregation (OLA) has been intro...
متن کاملAdaptive partitioning and indexing for raw data querying
Traditional database management systems approach to data analytics assumes that the input would be loaded within the DBMS, and then queried upon. However, data analytics depend on the interaction with the data analyst and as data collections grow larger and larger, data loading acts as a bottleneck and it incurs significant data-to-query delay. In this paper, we examine the NoDB paradigm, which...
متن کاملIn-Situ Processing and Visualization for Ultrascale Simulations
The growing power of parallel supercomputers gives scientists the ability to simulate more complex problems at higher fidelity, leading to many high-impact scientific advances. To maximize the utilization of the vast amount of data generated by these simulations, scientists also need scalable solutions for studying their data to different extents and at different abstraction levels. As we move ...
متن کاملIncentivizing Exploration In Reinforcement Learning With Deep Predictive Models
Achieving efficient and scalable exploration in complex domains poses a major challenge in reinforcement learning. While Bayesian and PAC-MDP approaches to the exploration problem offer strong formal guarantees, they are often impractical in higher dimensions due to their reliance on enumerating the state-action space. Hence, exploration in complex domains is often performed with simple epsilon...
متن کاملEffects of Heat Processing of Soybeans and Linseed on Ruminal Fatty Acid Biohydrogenation in situ
The aim of this study was to determine and compare in situ biohydrogenation (BH) fatty acids in three forms of soybeans and linseed (raw, extruded and roasted). Nylon bags (5×10 cm) containing 4 g of raw, extruded or roasted soybeans or raw, extruded or roasted linseed were incubated in the rumen of fistulated ewes for 4, 8, 12 and 24 hours. Results for linoleic acid (C18:2) showed tha...
متن کامل