Designing a hypothesis test to determine the best of two machine learning algorithms with only a small data set available is not a simple task. Many popular tests suffer from low power (5x2 cv [2]), or high Type I error (Weka’s 10x10 cross validation [11]). Furthermore, many tests show a low level of replicability, so that tests performed by different scientists with the same pair of algorithms...

An important task in machine learning is determining which learning algorithm works best for a given data set. When the amount of data is small the same data needs to be used repeatedly in order to get a reasonable estimate of the accuracy of the learning algorithms. This results in violations of assumptions on which standard tests are based and makes it hard to design a good test. In this arti...

The mining of software archives has enabled new ways for increasing the productivity in software development: Analyzing software quality, mining project evolution, investigating change patterns and evolution trends, mining models for development processes, developing methods of integrating mined data from various historical sources, or analyzing natural language artifacts in software repositori...

Social scientists envy the objectivity, controlled experimentation and replicability of hard sciences, a lack of which, they daim, hampers their ability to advance their disciplines and make them more useful and relevant to real life applications. This paper examines a specific area of social science, time series forecasting, which, through empirical studies using real-life data, allows for obj...

The general utility of large-scale artificial ecosystems for ecological and ecotoxicological research is evaluated by a case study of the replicability, repeatability and ecological realism of a Baltic Sea hard bottom littoral mesocosm (called BHB-mesocosm). The structure (species abundance and biomass listings) of the macrofauna community associated with bladder-wrack, Fucus vesiculosus L., in...