This can be done with stat = "identity"
, as mentioned in some of the comments, but the trick is to get the outliers into the data. Outliers need to be provided in a list column. Here's how you do it.
First, make up some data and draw a regular boxplot:
set.seed(123)
d <- data.frame(y = c(rnorm(100), rnorm(100)+.5, rnorm(100)-1),
x = rep(c("A", "B", "C"), each = 100))
ggplot(d, aes(x, y)) + geom_boxplot()

Now, calculate the stats manually and draw boxplot with alternative outlier definition. Note that I use mean +/- 2*SD so I get a few more outliers. It should be obvious how to change the code to +/- 4*SD.
library(dplyr)
d %>% group_by(x) %>%
summarize(middle = median(y),
mean = mean(y),
sd = sd(y),
lower = quantile(y, probs = .25),
upper = quantile(y, probs = .75),
ymin = max(mean - 2*sd, min(y)),
ymax = min(mean + 2*sd, max(y)),
outliers = list(y[y<ymin | y > ymax])) %>%
ggplot(aes(x, ymin = ymin, lower = lower,
middle = middle, upper = upper, ymax = ymax,
outliers = outliers)) +
geom_boxplot(stat = "identity")

Disclaimer: I've only tested this with the current development version of ggplot2, not sure whether it works with the version currently on CRAN.