How do I get msbuild /restore to work for a standalone/non-SDK project file?

Question

I want to use msbuild /restore with my project file. However, my project file is more like a script which orchestrates building multiple projects with particular properties, etc. Thus, it doesn’t make sense for me to set Sdk="Microsoft.NET.Sdk" because that causes weird errors to show up. However, if I don’t specify the Sdk, the /restore is ignored and it fails to actually restore anything.

Here is my example standalone project:

<?xml version="1.0" encoding="utf-8"?>
<Project DefaultTargets="Build" xmlns="http://schemas.microsoft.com/developer/msbuild/2003">
  <ItemGroup>
    <PackageReference Include="RoslynCodeTaskFactory" Version="2.0.7" />
  </ItemGroup>

  <Target Name="Build">
    <HelloWorld/>
  </Target>

  <UsingTask AssemblyFile="$(RoslynCodeTaskFactory)" Condition="'$(RoslynCodeTaskFactory)' != ''" TaskFactory="CodeTaskFactory" TaskName="HelloWorld">
    <Task>
      <Code Type="Fragment" Language="cs">
        <![CDATA[
          Console.WriteLine("Hello, world!");
        ]]>
      </Code>
    </Task>
  </UsingTask>
</Project>

My invocation and output:

C:\Users\binki\AppData\Local\Temp>"\Program Files (x86)\Microsoft Visual Studio\2017\Community\MSBuild\15.0\Bin\amd64\MSBuild.exe" /restore helloworld.proj
Microsoft (R) Build Engine version 15.7.177.53362 for .NET Framework
Copyright (C) Microsoft Corporation. All rights reserved.

Build started 2018-05-15 01:23:28.
Project "C:\Users\binki\AppData\Local\Temp\helloworld.proj" on node 1 (default targets).
C:\Users\binki\AppData\Local\Temp\helloworld.proj(8,5): error MSB4036: The "HelloWorld" task was not found. Check the following: 1.) The name of the task in the project file is the same as the name of the task class. 2.) The task class is "public" and implements the Microsoft.Build.Framework.ITask interface. 3.) The task is correctly declared with <UsingTask> in the project file, or in the *.tasks files located in the "C:\Program Files 9x86)\Microsoft Visual Studio\2017\Community\MSBuild\15.0\Bin\amd64" directory.
Done Building Project "C:\Users\binki\AppData\Local\Temp\helloworld.proj" (default targets) -- FAILED.


Build FAILED.

"C:\Users\binki\AppData\Local\Temp\helloworld.proj" (default target) (1:2) ->
(Build target) ->
  C:\Users\binki\AppData\Local\Temp\helloworld.proj(8,5): error MSB4036: The "HelloWorld" task was not found. Check the following: 1.) The name of the task in the project file is the same as the name of the task class. 2.) The task class is "public" and implements the Microsoft.Build.Framework.ITask interface. 3.) The task is correctly declared with <UsingTask> in the project file, or in the *.tasks files located in the "C:\Program Files (x86)\Microsoft Visual Studio\2017\Community\MSBuild\15.0\Bin\amd64" directory.

    0 Warning(s)
    1 Error(s)

Time Elapsed 00:00:01.71