Compute sample size and power for an ANOVA in Excel

This tutorial explains how to calculate the sample size and power for an ANOVA in Excel using XLSTAT.

What is the power of a statistical test?

XLSTAT offers features to apply analysis of variance (ANOVA), repeated measures analysis of variance and analysis of covariance (ANCOVA). For any of these methods, you may want to estimate the power or calculate the necessary number of observations before running the analysis.

When testing a hypothesis, there are several decisions to take:

• The null hypothesis H0 and the alternative hypothesis Ha.

• The statistical test to use.

• The type I error also known as alpha. It occurs when one rejects the null hypothesis when it is true. It is set a priori for each test and is 5%.

The type II error or beta is less studied but is of great importance. In fact, it represents the probability that one does not reject the null hypothesis when it is false. We can not fix it upfront, but based on other parameters of the model we can try to minimize it. The power of a test is calculated as 1-beta1−beta and represents the probability that we reject the null hypothesis when it is false.

We therefore wish to maximize the power of the test. XLSTAT calculates the power (and beta) when other parameters are known. For a given power, it also allows to calculate the sample size that is necessary to reach that power.

The statistical power calculations are usually done before the experiment is conducted. The main application of power calculations is to estimate the number of observations necessary to properly conduct an experiment.

Goal of this tutorial

We want to carry out an experiment where three types of microwaves were tested to explain the percentage of edible popcorn after cooking. Cooking differs depending on the brand of microwave, the power, and the duration. Here we will seek to determine whether there is a significant influence of brand, potency, cooking time, and possibly their interactions.

We are in the case of a 3-factor anova. The microwave brand has 3 levels, power and duration have two levels each. So we have 3 x 2 x 2 = 12 groups. In our case, we will seek to know the size of the sample necessary so that the bfactor is a power of 0.9.

As we do not know yet the parameters of our sample, we will use the concept of effect size. Cohen (1988) introduced this concept which makes it possible to give an order of magnitude for the importance of the effect. We will therefore test 3 sizes of effects: 0.1 for a weak effect, 0.25 for a moderate effect and 0.4 for a strong effect. It is expected that the larger the effect size, the smaller the required sample size will be.

Set up the sample size calculation for an ANOVA

Once XLSTAT has been launched, click on the Power icon and choose the ANOVA/ANCOVA function.

Once the button is clicked, the dialog box pops up.

You must then choose the objective Find sample size, and then select the ANOVA Factors and Interactions test.

The alpha is 0.05. The desired power is 0.9.

The number of groups, as explained earlier, is 12. The number of degrees of freedom of the numerator (Num DDL) is equal to the number of levels of the factor brand - 1, therefore 2.

In the Chart tab, the simulation plot option is activated and the size of sample 1 will be represented on the vertical axis and the power on the horizontal axis.

The power varies between 0.8 and 0.95 in intervals of 0.01.

Once you click on the OK button, the calculations begin and then the results are displayed.

Interpreting the results of sample size calculations for ANOVA

The first table gathers the parameters used as input.

The second table gathers the results of the calculation as well as an interpretation of the results.

We see that 83 observations are needed in our sample to obtain a power as close as possible to 0.9.

The following table gathers the calculations obtained for each value of the power between 0.8 and 0.95.

The simulation plot shows the evolution of the sample size as a function of the power. We see that for a power of 0.8, 64 observations per sample are enough and that for a power of 0.95 we arrive at 100 observations.

For effect sizes of 0.1 and 0.25, we obtain the following results:

The smaller the effect size, the more restrictive the input parameters will be and therefore the larger the sample size will be.

XLSTAT is therefore a powerful tool both for finding the sample size required for an analysis and for calculating the power of a test. Obviously, if one has more information about their sample, they can give details of the input parameters, rather than entering the size of the effect.