Why do HTML forms convert "&" into "&"?

Question

I've been trying to figure out why my carefully prepared "&" phrases were being turned into plain "&" phrases. I knew it was happening, but I didn't know if it was happening when they were being submitted as part of an SQL query or somewhere else. This is quite tricky, since you have to View Source to see the difference!

I eventually discovered where it was happening - in the HTML form that was being submitted (action="post"). I had a <select> where one of the options contained the phrase:

<option value="sticks &amp; stones">sticks &amp; stones</option>

I found that when the form was submitted, the value had been changed to "sticks & stones", with the result that when the value was submitted in a database query, it failed to find any results.

I have further experimented and find this happens with text inputs and hidden inputs too.

My question is: WHY????? It seems a particularly silly thing to do.

Answer 1

Escape sequences have to be interpreted without knowing the author's intent. If I wanted a value like Foo " Bar, I couldn't say value="Foo " Bar" because the quotes wouldn't match. Instead, I'd have to use value="Foo " Bar". But then what if I literally want Foo " Bar? That's where & comes in. But to avoid ambiguity, the system has to always translate escape sequences. So if you want a literal &, you have to be explicit about it like sticks & stones.