Labels don't center with horizontal stacked bar (geom_col and position_fill)

Question

I want to create a horizontal stacked bar plot with geom_col. I can get it almost right, exact that the labels are not centered. I tried different combinations of aes arguments on the geom_text function, but up until now, no success. This is the closest I get. Anyone know what goes wrong?

df <- data.frame(year=c("2014", "2015", "2014", "2015", "2014", "2015", "2014", "2015"),
                 city=c("Berlin", "Berlin", "Paris", "Paris", "New York", "New York", 
                        "Frankfurt", "Frankfurt"),
                 number=c(501,2285,558, 4001, 600, 3000, 305, 35000))

fig <- ggplot(df, aes(x = year, y = number)) +  
  geom_col(aes(fill = city), position = position_fill()) +
  geom_text(aes(y = number, label = number, group = year), 
            position=position_fill(vjust = 0.5), colour = "white") +
  coord_flip() +
  theme_bw() +
  theme(legend.position = "none") +
  scale_y_continuous(breaks = c(0,0.2,0.4,0.6,0.8,1.0), 
                     labels = c("0%","20%","40%","60%","80%","100%"))
fig