Parallel Reconstruction of CLEO III Data

Reconstruction of one run of CLEO III raw data can take up to 9 days to complete using a single processor. This is an administrative nightmare, and even minor failures result in reprocessing the entire run, which wastes time, money and CPU power. We leveraged the ability of the CLEO III software infrastructure to read and write multiple file formats to perform reconstruction of a single run using several CPUs in parallel. Using the Sun Grid Engine and some Perl scripts, we assign roughly equal-sized chunks of events to different CPUs. The Raw data are read from an Objectivity/DB database, but the reconstruction output is written to a temporary file, not the database. This takes about 6 hours. Once all the chunks have been analyzed, they are gathered together in event-number order and injected into Objectivity/DB. This process takes an additional 6 to 18 hours, depending on run size. A web-based monitoring tool displays the status of reconstruction. Many benefits accrue from this process, including a dramatic increase in efficiency, a 20% increase in luminosity processed per week, more predictable and manageable processor farm load, reduction in the time to stop the processor farm from up to 9 days to less than 24 hours, superior fault tolerance, quicker feedback and repair times for bugs in the reconstruction code, and faster turnaround of early runs for data quality and software correctness checks.