Synchronized performance evaluation methodologies for communication systems

Researchers and developers typically rely on analytical techniques, simulation-based approaches or real-world experiments for the performance evaluation of communication systems. All of these methods have unique strengths. Network simulations are well suitable for conducting virtual experiments of very high scale; the deterministic and thus repeatable execution of simulations also make them an essential method for parameter studies. Performance evaluations carried out with real-world systems allow for investigations with the topmost possible level of realism and detail. Finally, analytical techniques abstract away from the technical environment of a communication system and hence enable the deduction of general evidence. Unfortunately, all performance evaluation techniques also suffer from a number of individual shortcomings. The analytical formalization of complex communication protocols and their behavior is often very difficult. Network simulations model only the essential functionality of a communication system, making it problematic to apply this technique for analyzing resource usage or system-specific effects. Finally, evaluations carried out with real-world software prototypes are often perturbed by uncontrollable conditions in the environment. In addition, the high amount of hardware and manpower required for real-world performance evaluations of larger scale make such trials often very costly. Hybrid performance evaluation methodologies are a promising approach for overcoming these issues. The concept of network emulation combines the flexibility and scalability of network simulation with the credibility and the level of detail associated with performance evaluations using real-world systems. A second important hybrid methodology is hardware/network co-simulation. The idea of this methodology is to integrate different simulation tools to bridge between their individual application domains, for instance network simulation and hardware modeling. We contribute to the field of hybrid evaluation tools for communication systems in different ways. First, we introduce the concept of Synchronized Hybrid Evaluation that generalizes the idea of hybrid performance evaluation and thus is able to subsume a number of existing hybrid techniques. The second central cornerstone of our work is the synchronization and the virtualization of time. Existing network emulation frameworks require real-time capable simulations, which limits the application of network emulation to a rather narrow set of scenarios. Our work on Synchronized Network Emulation removes this burden by synchronizing the execution of the network simulation with virtualized communication systems. A further contribution to the field of network emulation is our work on Device Driver-enabled Wireless Network Emulation. It tightly integrates the simulated environment with the operating system context of the software prototype. As we further elaborate in this dissertation, both concepts and their respective implementations substantially extend the applicability of network emulation. In addition, we contribute to the field of hybrid evaluation techniques with two additional frameworks. First, we propose the integration of a SystemC-based hardware simulator with a network simulation tool, aiming at a network centric design of embedded systems. Second, we show that virtualizing the progression of time and synchronizing the execution of virtualized software prototypes is helpful for building distributed debugging and monitoring tools.