I've noticed that many books built using RBookdown will have an "on this page" section in the upper right hand corner (displays book subsections). You can see what I'm referring to by looking at Hadley Wickham's "R4DS" book here: https://r4ds.had.co.nz/data-visualisation.html.
After digging through the source code I can't figure out how to create this section. Does anyone know how this section is created?
They are using a new document format bookdown::bs4_book. This new format renders the section in question. See the rdrr.io man page for some more information.
To use the new format, first install the latest bookdown version from github with remotes::install_github("rstudio/bookdown"). Then you can specify the new format for your bookdown project in your _output.yaml file like they have done here.
Note, there are a number of dependencies required to use the new format including downlit, bslib, and notably a version of htmltools >= 0.5.0.9001. downlit can be installed with remotes::install_github("r-lib/downlit") while bslib can be installed with remotes::install_github("rstudio/bslib"), and one would normally be able to install htmltools with just remotes::install_github("rstudio/htmltools"). However, as of this writing the latest version of htmltools is failing its R-CMD-Check, and I was unable to install it on my system. Therefore, I installed the 0.5.0.9001 version by referencing the requisite commit with remotes::install_github("rstudio/htmltools@8c82cc4f869f75ac4f13ee78ab322790ec316d3f").
