How could we visualize the explained variance?

Visualizing variance, and even more so explained variance, is a tough attempt. A very nice visualization of explained variance was developed by W.J. Schneider here, but unfortunately it does not cover the explained variance in an ANOVA example. Thus, I started with an own visualization of the data that we already have so far.

df <- df %>% 
  # group_by(group) %>%  # actually, we do not need the within variance, but total variance 
  mutate(grandmean = mean(value)) %>%
  mutate(diff = value - grandmean) %>%
  group_by(group) %>%
  mutate(groupmean = mean(value)) %>%
  ungroup()

pd = position_dodge(.01)
ggplot(df, aes(x=0, y=value, col=group))+
  geom_point(alpha=.3, pos=pd, size=1.7)+
  geom_hline(size=1.1, aes(yintercept=grandmean), linetype="dashed")+
  xlim(-.2,9)+
  coord_fixed()+
  theme_bw()+
geom_text(aes(x=7.3, y=0.6), label="Grand Mean",size=3.8, col="black", 
          fontface="bold")

We can plot the data from both groups on the same line and calculate the mean of all values together, the so called Grand Mean. That would be the mean across all groups. The total variation of the values is the deviation of all values from this Grand Mean. This total variation can be decomposed into systematic variation — how values vary systematically between the groups — as well as into unsystematic variation — how values vary randomly within a group (i.e. error variance).

The variance for the population is defined as: $${\hat{\sigma }}^2=\frac{(x_i-\hat{\mu }{)}^2}{N-1}=\frac{SS}{N-1}$$ You might know that SS stands for Sum of Squares, and that it is possible to visualize them by squaring the residuals; e.g. from a regression line. By drawing the distance between a value and the mean, and then squaring this term, we get these Sums of Squares. The squaring can be shown by adding a line that is perpendicular to the first and has the same length. Let’s add these squares:

ggplot(df, aes(x=0, y=value, col=group))+
  geom_rect( aes(ymin=grandmean, ymax =value, xmin = 0, xmax=0+abs(diff)), 
             color="black",fill="darkgray", alpha=.2) +
    geom_point(alpha=.3, pos=pd, size=1.7)+
    geom_hline(size=1.1, aes(yintercept=grandmean), linetype="dashed")+
  theme_bw()+
  coord_fixed()  

The area of all those squares would be the total Sum of Squares as a measure for the total variation. Let’s now add the systematic variation. First we need to calculate the mean of every group. The systematic variation is the difference from these means to the Grand Mean (multiplied by group size). We can visualize this systematic variation in a similar way:

ggplot(df, aes(x=0, y=value, col=group, fill=group))+
  geom_rect( aes(ymin=grandmean, ymax =value, xmin = 0, xmax=0+abs(diff)), color="black", fill="darkgray",alpha=.2) +
    geom_hline(size=1.1, aes(yintercept=grandmean), linetype="dashed")+
    geom_point(alpha=.3, pos=pd, size=1.7)+
  theme_bw()+
  coord_fixed()+
    geom_rect(aes(ymin=grandmean, ymax=groupmean, xmin=0, xmax=0+abs(grandmean-groupmean)), color="black",alpha=.06) +
  geom_segment(aes(y=groupmean, yend=groupmean, x=0, xend=5), size=1.4, linetype="longdash" )

At this moment, there is a lot of overlap of the squares, and also the multiplication of the systematic variation (SS_between) with the group size is not visible. Thus, we will now add up all the squares and plot them beside each other.

Altough there is a lot of overlap on the boxes, and we have to imagine that the systematic variation gets multiplied by group size, we see already, that the area of systematic variation is much smaller than the area of the total variation. This can also be achieved mathematically and would lead us to the following ratio of between variance to total variance:

areas <- df %>%
  ungroup() %>%
  mutate(diff_2 = diff^2) %>%
  summarise(area=sum(diff_2)) # total variation
total <- sum(df$diff^2)

# SS between is the squared difference of groupmeans to grandmean multiplied by group size
grandmean <- unique(df$grandmean)
between <- ( (mean(mixed)-grandmean)^2 + (mean(same)-grandmean)^2  ) * n

between/total

## [1] 0.04372711

Which leads us again exactly to our calculated $\eta$². The relation of the area would look like the following:

ggplot( ) +
  theme_bw()+
  coord_fixed()+
  geom_rect(aes(xmin=0, xmax=sqrt(total),ymin=0, ymax=sqrt(total)), color="darkgray", alpha=.2)+
  geom_rect(aes(xmin=0, xmax=sqrt(between), ymin=0, ymax=sqrt(between)),fill ="skyblue",alpha=.6)+
  geom_text(aes(x=3.8,y=4), label="explained variance", size=3.2, col="darkblue", fontface="bold") +
  geom_text(aes(x=18, y=19), label="unexplained variance",size=7.8, col="#999999", fontface="bold")

So what on earth does eta squared tell us?

We have now visualized $\eta$² in terms of the explained variance, but still it is not clear, how to interpret the values of $\eta$² and why they do differ so much from other effect sizes like Cohen’s d. To develop a feeling for the “behavior” of $\eta$², I simulated data for two groups with different mean differences and sample sizes with the research question: “How is $\eta$² distributed across different effects?” I simulated 1000 datasets with varying sample sizes (N=30, 50, 100, 250, 500) and varying mean differences (Cohen’s d = 0.00, 0.10, 0.30, 0.50, 1.00) to investigate the relation between Cohen’s d and $\eta$².

final_df <- data.frame(d=numeric(25000), n=numeric(25000), i=numeric(25000), Fval=numeric(25000), sign=numeric(25000), eta=numeric(25000), cohen=numeric(25000))

count <- 1
for(d in c(0,.1,.3,.5,1)){ # vary mean difference (cohen's d)
  for(n in c(30,50,100,250,500)){ # vary sample size of both experimental groups
    for( i in 1:1000){ # 1000 times
  

  KG <- rnorm(n, mean=0, sd=1) #simulate sample from a standard normal distribution
  EG <- rnorm(n, mean=0+d, sd=1)

  temp_df <-data.frame(group=c(rep("KG",n),rep("EG",n)), value=c(KG, EG))
  mod2 <- aov(temp_df$value ~ temp_df$group)
  Fval <- unlist(summary(mod2))["F value1"]
  sign <- unlist(summary(mod2))["Pr(>F)1"]
  eta <- etaSquared(mod2)[1]
  cohen <- CohenD(EG,KG)[1]
  
  final_df[count,] <- c(d,n,i,Fval, sign, eta, cohen)
 count <- count+1
    }}}

#extract the following: d, n, i, F, significance, eta,  cohenD

ggplot(final_df, aes(x=cohen, y=eta, color=Fval))+
      #geom_jitter(width=.01)+
      geom_point(alpha=.2)+
      facet_wrap(~n)+
      scale_color_viridis_c(name="F value")+
      ylab("eta squared")+
      #ylim(-1,2)+ coord_fixed()+
      #facet_wrap(~n, scales="free_x")+  #
      theme_bw()

  # ggplot(final_df, aes(x=Fval, y=eta))+
  #   geom_point()+
  #   geom_smooth()+
  #   facet_wrap(~n)+ #very interesting!
  #   theme_bw()  # non-linear relation?
  # 
  # ggplot(final_df, aes(x=n, y=eta))+
  #   geom_point()  #interesting

We can see a strong relationship between Cohen’s d and $\eta$² that seems to be linear except when it is near 0 — the u-shape comes from the fact, that while Cohen’s d can take negative values, eta squared is always positive, and gets larger for values of d that are farther away from 0. We can also see, that Cohen’s d is normally larger than $\eta$². Furthermore, there is an upper bound of eta, that is far lower than 1, which would be the logical maximum. In fact, $\eta$² would be 1, only if all values within a group are the same (e.g. in the mixed-gender condition, everyone nominates 3 members, in the same-gender condition, everyone nominates 2). If the values are not all the same, but normally distributed, the upper bound for $\eta$² is at .64 (Pedhazur, 1997 in Richardson, 2011). In our simulation $\eta$² hardly ever exceeds .5, and its average value for the different sample sizes and Cohen’s d is given in the next table.

tb <- final_df %>% 
  dplyr::group_by(d,n) %>% 
  dplyr::summarise(mean=mean(eta)) %>%
  mutate(mean=round(mean,3)) %>%
  spread( n, mean)

tb %>%
  kable() %>%
  kable_styling(bootstrap_options = "striped") %>%
  column_spec(1, bold = T, border_right = T) %>%
  add_header_above(c("Cohen's d", "Sample size (N)" = 5))

Interestingly, the mean and upper bound of $\eta$² seem to decrease with larger samples. While in large samples the F value easily exceeds 100, which is not the case in samples with 30 or 50 participants per group, Cohen’s d as well as $\eta$² have a smaller range in larger samples. Thus, the u-shaped curve for N of 500 looks like a snippet of the curve for an N of 30, as the variation is larger in smaller samples.

When there is no true effect, $\eta$² might still yield values that would be counted as an effect. For example, when comparing it with the rule of thumb, that $\eta$² = .06 would mean a medium effect, in 10.29% of the cases, $\eta$² would count as a medium effect, while the true effect is zero; but this is mostly the case in smaller samples.

So far, we have now investigated how Cohen’s d and $\eta$² are related in a unifactorial ANOVA with two groups. We have seen that there is a strong relationship, but also that $\eta$² is smaller than Cohen’s d and that it has an upper bound. To help researchers to compare their values of eta squared against some common magnitudes, also some benchmarks have been developed, that suggest, what counts as a small, medium and large effect.

Concluding remarks

We have now investigated the behavior of $\eta$² and compared it to other effect size measures. We have seen, that $\eta$² differs quite a lot from other effect sizes in magnitude, and we visualized what it means in terms of variance decomposition. Note, that these scenarios above only considered ANOVAs with one independent variable/factor. Estimating the (partial) $\eta$² for ANOVAs with multiple factors or with covariates (ANCOVAs) is more complex and not discussed here, because variance decomposition in those cases is less clear (Richardson, 2011). This makes it also difficult, to compare results from various designs.

As Richardson (2011) pointed out — despite its wide use — the interpretation of $\eta$² as well as partial $\eta$² has to be done carefully. First of all, there has been a vast confusion about $\eta$² and partial $\eta$², as for example SPSS did not use the correct term up to version 10. Furthermore, partial $\eta$² depends on the inclusion and exclusion of other factors. Secondly, the term explained variance might also be misleading, as we did not explain anything with this measure. It is simply a ratio, that tells us, what proportion in the variation of the dependent variable is associated with falling in one of the groups of the independent variable (Richardson, 2011).

References

• Cohen, J. (1969). Statistical power analysis for the behavioural sciences. New York: Academic Press.
• Correll, J., Mellinger, C., McClelland, G. H., & Judd, C. M. (2020). Avoid Cohen’s ‘Small’,‘Medium’, and ‘Large’for Power Analysis. Trends in Cognitive Sciences.
• Hays, W. L. (1963). Statistics. New York: Holt, Rinehart, & Winston.
• Huguet, P., & Regner, I. (2007). Stereotype threat among schoolgirls in quasi-ordinary classroom circumstances. Journal of educational psychology, 99(3), 545.
• Kelley, T. L. (1935). An unbiased correlation ratio measure. Proceedings of the National Academy of Sciences, 21, 554–559. - Leonhart, R. (2009). Lehrbuch Statistik–Einstieg und Vertiefung (2., überarbeitete und erweiterte Aufl.). Bern: Verlag Hans Huber, 92.
• Lüdecke, M. D. (2019). Package ‘sjstats’.
• Olejnik, S., & Algina, J. (2003). Generalized eta and omega squared statistics: Measures of effect size for some common research designs. Psychological Methods, 8, 434–447.
• Schäfer, T., & Schwarz, M.A. (2019). The meaningfulness of effect sizes in psychological research: differences between sub-disciplines and the impact of potential biases. Frontiers in Psychology, 10, 1-13.
• Richardson, J. T. (2011). Eta squared and partial eta squared as measures of effect size in educational research. Educational Research Review, 6(2), 135-147.