That is because the original array is not modified unless you modify it yourself in the callback function.
Your callback function is basically:
function($item, $key) {
htmlspecialchars($item);
}
So while the function is called, nothing is stored and the original array is not changed.
If you want to modify the value in the function, you can pass it by reference:
function(&$item, $key) {
$item = htmlspecialchars($item);
}
So the result would look like:
$new[] = "<a href='test'>Test</a><li><div>";
array_walk_recursive($new, function(&$item, $key) {
$item = htmlspecialchars($item);
});
var_dump($new) ; // change!
You can of course define a separate function if you would prefer that.