ggplot: How can I add a colored bar to the side of a plot based on the levels of a factor?

Question

I have a plot and I am trying to add a simple bar plot next to the x and y margins with a legend to indicate grouping of the x and y axis.

data <- read.table(text = "Type Category Value GroupX GroupY
 S1 A   73 1 1
 S2 A   57 1 2
 S1 B   7 1 2
 S2 B   23 2 2
 S1 C   51 3 3
 S2 C   87 3 3", header=TRUE)

I can create the plot like this:

library(ggplot2)

g = ggplot(aes(x=Type, y=Category, fill=Value), data=data)+ 
  geom_tile() + 
  scale_fill_gradient2(low="#D7191C", mid="white", high="#2C7BB6") +
  scale_colour_gradient(low="grey30", high="white", guide="none") + 
  labs(y=NULL, x=NULL, fill="Value") + 
  geom_vline(xintercept=1.5, size=1.5, color="grey50") + 
  theme_bw() +
  theme(axis.text.x=element_text(angle = -45, hjust = 0))

And then add a bar plot to the y-axis for example like this:

g+annotate("rect", xmin=0, xmax=0.5, ymin=-Inf, ymax=Inf, fill="green")

But I would like to "fill" it with a stacked colored bar depending on the values of data$GroupY, how can I do this?

g+annotate("rect", xmin=0, xmax=0.5, ymin=-Inf, ymax=Inf, fill=data$GroupY)

Thank you for the help!