Color dataset by group and add geom_vline to legend only

Question

I have a genome-wide dataset that I'm trying to plot in the following way:

1. Have each chromosome be a separate color
2. Have specific windows highlighted by a bar (I'm using geom_vline) - this I'm getting from a separate table
3. Have only geom_vline feature in the legend

I have tried many different things, but it seems I cannot have all three together!

Here is the link to both datasets: allStats & allStats_fstPi_group15

With this code, I can have the first 2, but not the 3rd:

ggplot(allStats, aes(x = mid2, y = Fst_group1_group5,
                     color = as_factor(scaffold))) +
  geom_point(size = 2) +
  geom_vline(xintercept = chrom$add, color = "grey") +
  scale_y_continuous(expand = c(0,0), limits = c(0, 1)) +
  scale_x_continuous(labels = chrom$chrID, breaks = axis_set$center) +
  scale_color_manual(values = rep(c("#276FBF", "#183059"), unique(length(chrom$chrID)))) +
  scale_size_continuous(range = c(0.5,3)) +
  labs(x = NULL, 
       y = "Fst SBM vs OC") + 
  theme_minimal() +
  theme( 
    legend.position = "none",
    panel.grid.major.x = element_blank(),
    panel.grid.minor.x = element_blank(),
    panel.grid.major.y = element_blank(),
    panel.grid.minor.y = element_blank(),
    axis.title.y = element_text(),
    axis.text.x = element_text()) +
  geom_vline(data = allStats_fstPi_group15,
             aes(xintercept = allStats_fstPi_group15$mid2),
             color = "orange", show.legend = T)

With this one I can get 2 and 3 only (I'm not able to color code each block separately):

cols <- c("SBM vs OC" = rep(c("#276FBF", "#183059"), unique(length(chrom$chrID))), 
          "90th percentile (Fst vs Pi)" = "orange")

ggplot(allStats, aes(x = mid2)) +
  geom_point(aes(y = Fst_group1_group5, 
                 color = as_factor(scaffold)), 
             size = 2) +
  geom_vline(data = allStats_fstPi_group15,
             aes(xintercept = allStats_fstPi_group15$mid2, 
                 color = "90th percentile (Fst vs Pi)")) +
  scale_color_manual(values = cols)

I've seen the issue with the legend being that color needs to be within aes(), so my question is: is it impossible what I'm trying to do?