I tried to use your code to accomplish this, in fact, your code is working and the file can be downloaded, but the download prompt box does not appear.
When you open your browser's download list, the file should appear in your list.

I use JavaScript to call this method, the download prompt box appears normally and the file can be downloaded.
Below is my test code,you can refer to it:
Controller:
public void GenerateXML()
{
XmlDocument doc = new XmlDocument();
// XML declaration
XmlNode declaration = doc.CreateNode(XmlNodeType.XmlDeclaration, null, null);
doc.AppendChild(declaration);
// Root element: Catalog
XmlElement root = doc.CreateElement("Catalog");
doc.AppendChild(root);
// Sub-element: srsapiversion of root
XmlElement book = doc.CreateElement("book");
root.AppendChild(book);
// Sub-element: srsapiversion of root
XmlElement bookname = doc.CreateElement("bookname");
book.InnerText = "2 States";
root.AppendChild(bookname);
// Sub-element: id of root
XmlElement id = doc.CreateElement("id");
id.InnerText = "70-515";
root.AppendChild(id);
// Sub-element: author of root
XmlElement author = doc.CreateElement("author");
author.InnerText = "Chetan Bhagat";
//Attribute age of author
XmlAttribute age = doc.CreateAttribute("age");
age.Value = "43";
author.Attributes.Append(age);
root.AppendChild(author);
System.IO.MemoryStream stream = new System.IO.MemoryStream();
XmlTextWriter writer = new XmlTextWriter(stream, System.Text.Encoding.UTF8);
doc.WriteTo(writer);
writer.Flush();
Response.Clear();
byte[] byteArray = stream.ToArray();
Response.Headers.Add("Content-Disposition", "filename=Books.xml");
Response.Headers.Add("Content-Length", byteArray.Length.ToString());
Response.ContentType = "application/octet-stream";
Response.Body.Write(byteArray);
writer.Close();
}
JavaScript:
<script>
window.open('/Home/GenerateXML', 'DownloadWindowName');
</script>
Test Result:

Hope this will help you.