R plot 'Heat map' of set of draws

Question

I have a matrix with x rows (i.e. the number of draws) and y columns (the number of observations). They represent a distribution of y forecasts.

Now I would like to make sort of a 'heat map' of the draws. That is, I want to plot a 'confidence interval' (not really a confidence interval, but just all the values with shading in between), but as a 'heat map' (an example of a heat map ). That means, that if for instance a lot of draws for observation y=y* were around 1 but there was also a draw of 5 for that same observation, that then the area of the confidence interval around 1 is darker (but the whole are between 1 and 5 is still shaded).

To be totally clear: I like for instance the plot in the answer here, but then I would want the grey confidence interval to instead be colored as intensities (i.e. some areas are darker).

Could someone please tell me how I could achieve that?

Thanks in advance.

Edit: As per request: example data. Example of the first 20 values of the first column (i.e. y[1:20,1]):

[1]  0.032067416 -0.064797792  0.035022338  0.016347263  0.034373065 
0.024793101 -0.002514447  0.091411355 -0.064263536 -0.026808208 [11]  0.125831185 -0.039428744  0.017156454 -0.061574540 -0.074207109 -0.029171227  0.018906181  0.092816957  0.028899699 -0.004535961

Answer 1

So, the hard part of this is transforming your data into the right shape, which is why it's nice to share something that really looks like your data, not just a single column.

Let's say your data is this a matrix with 10,000 rows and 10 columns. I'll just use a uniform distribution so it will be a boring plot at the end

n = 10000
k = 10
mat = matrix(runif(n * k), nrow = n)

Next, we'll calculate quantiles for each column using apply, transpose, and make it a data frame:

dat = as.data.frame(t(apply(mat, MARGIN = 2, FUN = quantile, probs = seq(.1, 0.9, 0.1))))

Add an x variable (since we transposed, each x value corresponds to a column in the original data)

dat$x = 1:nrow(dat)

We now need to get it into a "long" form, grouped by the min and max values for a certain deviation group around the median, and of course get rid of the pesky percent signs introduced by quantile:

library(dplyr)
library(tidyr)
dat_long = gather(dat, "quantile", value = "y", -x) %>%
    mutate(quantile = as.numeric(gsub("%", "", quantile)),
           group = abs(50 - quantile))

dat_ribbon = dat_long %>% filter(quantile < 50) %>%
    mutate(ymin = y) %>%
    select(x, ymin, group) %>%
    left_join(
        dat_long %>% filter(quantile > 50) %>%
        mutate(ymax = y) %>%
        select(x, ymax, group)
    )

dat_median = filter(dat_long, quantile == 50)

And finally we can plot. We'll plot a transparent ribbon for each "group", that is 10%-90% interval, 20%-80% interval, ... 40%-60% interval, and then a single line at the median (50%). Using transparency, the middle will be darker as it has more ribbons overlapping on top of it. This doesn't go from the mininum to the maximum, but it will if you set the probs in the quantile call to go from 0 to 1 instead of .1 to .9.

library(ggplot2)
ggplot(dat_ribbon, aes(x = x)) +
    geom_ribbon(aes(ymin = ymin, ymax = ymax, group = group), alpha = 0.2) +
    geom_line(aes(y = y), data = dat_median, color = "white")

Worth noting that this is not a conventional heatmap. A heatmap usually implies that you have 3 variables, x, y, and z (color), where there is a z-value for every x-y pair. Here you have two variables, x and y, with y depending on x.

Answer 2

That is not a lot to go on, but I would probably start with the hexbin or hexbinplot package. Several alternatives are presented in this SO post.

Formatting and manipulating a plot from the R package "hexbin"