I don't have the platform available to give you a proper walkthrough on setting this up, but I should be able to get you in the right place. First of all, do not make changes in the app/code/core files. Any changes you would absolutely need to make to those files you should do by making a copy of lets say app/code/core/Mage/Sales/something.php to app/code/local/Mage/Sales/something.php. Magento knows to automatically use the code in local to override the code in core.
If you take a look at the source code for that page you'll see where the name form is:
<input id="name" name="product[name]" value="" class=" required-entry input-text required-entry" type="text"/> </td>
<td class="scope-label"><span class="nobr">[STORE VIEW]</span></td>
</tr>
What's going on here is you'll notice under class we have "required-entry input-text and, well, required-entry again. These are the validation tags defined in js/prototype/validation.js. You will need to add some custom validation to it, and add it to your template file (not in core, it can break when you upgrade).
You'll notice in validation.js a section
Validation.add('IsEmpty', '', function(v) {
In this section you can add your custom validation. Lets say:
//make sure these are unique, I'm not checking
['validate-length', 'Your input needs to be less than x characters.', function(v) {
if (v.length > x) return false;
}],
If you need help finding the template location, take a look at: Finding Correct Templates and Blocks in Magento. You'll simply add validate-length class such as: class="required-entry validate-length..."