Making patchwork compatible with pdf creator using downloadHandler in Shiny?

Question

I'm interested in using the patchwork package in arranging a set of plots, which the end user can then download using a download button. For simplicity I've only shown one plot in this example.

Sample script:

library(shiny)
library(tidyverse)
library(Cairo)
library(grDevices)

ui <- fluidPage(

  mainPanel(
    tabsetPanel(
      tabPanel(
        "PDF this plot with patchwork", 
        plotOutput("diamonds"),
        downloadButton("downloadPlot", "Download Plot")
      )
    )
  )
)

server <- function(input, output, session){

  output$diamonds <- renderPlot({
    ggplot(diamonds, aes(x=carat, y=price, color=cut)) + geom_point() + geom_smooth()
  })

  output$downloadPlot <- downloadHandler(
    filename = "my_plot.pdf",

    content = function(file){
      cairo_pdf(filename = file,
                width = 18, height = 10, pointsize = 12, family = "sans", bg = "transparent",
                antialias = "subpixel",fallback_resolution = 300)
      patchwork::wrap_plots(ggplot(diamonds, aes(x=carat, y=price, color=cut)) + geom_point() + geom_smooth())
      dev.off()
    },

    contentType = "application/pdf"
  )
}

shinyApp(ui=ui, server=server)

In this case, the pdf generated is blank. I looked at this relevant question, and if I replace patchwork::wrap_plots with gridExtra::grid.arrange, the app now works, however there are a lot of reasons I would prefer to use patchwork which I haven't featured (plot naming, subtitles etc). For my real script I'm not interested in using Markdown either.