You must implement IDockingWindow. Result will be look like this:

One of the scenario:
1) Create BHO with IObjectWithSite support.
2) Inside IObjectWithSite.SetSite call Site.QueryInterface(IServiceProvider, ServiceProvider)
3)Then call ServiceProvider.QueryService(SID_SShellBrowser, IDockingWindowFrame, DockingWindowFrame).
4) Then call DockingWindowFrame.AddToolbar with your object that implements IDockingWindow and IObjectWithSite.
5) Inside IObjectWithSite of your object call QueryInterface(IDockingWindowSite, DockingWindowSite)
Other detail you can find in the MSDN.