Add significance star to boxplot when we have 2 grouping variables on the X axis

Question

Can you please help me create this typical plot? It is no big deal to get things done when we want to compare between two species of the iris data, but when we add another grouping variable, things don’t seem to work.

I want to compare between v.small and small sepal lengths of the sl_class variable but only for versicolor species.

I have tried this:

Code

# Load libraries 
library(dplyr)
library(ggplot2)
library(ggsignif) 

# Load iris dataset
data(iris)

# Create a new variable sl_class based on quartiles of Sepal.Length column
iris$sl_class <- cut(iris$Sepal.Length, 
                      breaks = quantile(iris$Sepal.Length, 
                      probs = c(0, 0.25,0.5, 0.75, 1)),
                      include.lowest = TRUE,
                      labels = c("v.small", "small", "big", "v.big"))

# boxplot
p <- ggplot(filter(iris, Species %in% c("setosa", "versicolor")), 
       aes(x= Species, y= Sepal.Length, color=sl_class)) +
     geom_boxplot()

# add significance star between v.small and small for versicolor
p + geom_signif(comparisons = list(c("v.small", "small"), test = "t.test") ) 

And got this warnig from R with no significance * or p-value on the plot.

Warning message:
Computation failed in `stat_signif()`
Caused by error in `if (scales$x$map(comp[1]) == data$group[1] | manual) ...`:
! missing value where TRUE/FALSE needed