I did something similar in the past. The code is in VB but it's very simple and you can easily convert it to C#.
Your code to build the project is OK. But if you have more Platforms available, such as "x86" and "Any CPU", the SolutionBuild.BuildProject takes a random platform. So it's better to explicitly specify it, like:
dte.Solution.SolutionBuild.BuildProject("Debug|Any CPU", project.UniqueName);
You can get the current configuration name and platform from:
Dim solutionConfiguration As SolutionConfiguration2 = CType(dte.Solution.SolutionBuild.ActiveConfiguration, SolutionConfiguration2)
Then you cannot start debugging particular method. DTE can only start to debug the startup object, e.g. Main. Exactly as if you press F5. If you want to start any particular method, it is more difficult.
You wrote, you can instantiate and call the method with reflection. This is OK. But remember, you cannot debug the process itself. This is how Windows works. It means, Visual Studio process can only debug ANOTHER process. So you cannot instantiate your compiled assembly in VSIX process (which is in fact a VS process).
You must create a new process and load your assembly in it. I don't have a code how to do it. Just for your information, the path of the compiled assembly can be retrieved like this:
Private Shared Function GetAssemblyPath(proj As EnvDTE.Project) As String
Dim fullPath As String = proj.Properties.Item("FullPath").Value.ToString()
Dim outputPath As String = proj.ConfigurationManager.ActiveConfiguration.Properties.Item("OutputPath").Value.ToString()
Dim outputDir As String = Path.Combine(fullPath, outputPath)
Dim outputFileName As String = proj.Properties.Item("OutputFileName").Value.ToString()
Dim assemblyPath As String = Path.Combine(outputDir, outputFileName)
Return assemblyPath
End Function
When you have the process and its ID, you must attach Visual Studio to this process:
Friend Sub AttachDebuggerToProcess(processId As Integer)
Dim dte As DTE2 = TryCast(Package.GetGlobalService(GetType(SDTE)), DTE2)
If dte Is Nothing Then
Return
End If
Dim proc As Process2 = Nothing
For Each process As Process In dte.Debugger.LocalProcesses
If process.ProcessID = processId Then
proc = process
Exit For
End If
Next
If proc Is Nothing Then
' report error
Return
End If
Try
If Not proc.IsBeingDebugged Then
proc.Attach()
End If
Catch exx As Exception
End Try
End Sub
And only now you can call your method with reflection. If there's a breakpoint set, it will be hit.