The huge difference between the Factory and Constructor tests is due to a series of extra steps performed in the latter.
When a function is invoked as a constructor, its [[Construct]] internal method is invoked, and that's likely the cause of the performance difference. Look at all the steps involved in [[Construct]]:
1. Let obj be a newly created native ECMAScript object.
2. Set all the internal methods of obj as specified in 8.12.
3. Set the [[Class]] internal property of obj to "Object".
4. Set the [[Extensible]] internal property of obj to true.
5. Let proto be the value of calling the [[Get]] internal property of F with argument "prototype".
6. If Type(proto) is Object, set the [[Prototype]] internal property of obj to proto.
7. If Type(proto) is not Object, set the [[Prototype]] internal property of obj to the standard built-in Object prototype object as described in 15.2.4.
8. Let result be the result of calling the [[Call]] internal property of F, providing obj as the this value and providing the argument list passed into [[Construct]] as args.
9. If Type(result) is Object then return result.
10. Return obj.