There are some debuggers for Firefox extensions, inparticular Venkman and Chromebug (which only works reliably with Firefox v5).
These will be able to break (pause code execution and launch debugger) when an exception occurs but don't usually help when you've got packaging problems, which are often caused by
- Bad manifests - look though Mozilla's packaging documentation, in particular the format of manifests.
- Errors in XML - validate all XML files (such as install.rdf).
- Referenced files that are missing (e.g. a DTD referenced in the XML that is missing)
With the lack of tools, the best option is often to compare your extension's manifests and directory structure against a known good extension (e.g. an example Hello World extension, or an earlier version of your extension) as something as simple as a wrong entity in a DTD file can cause the error you are seeing.
Is this is a classic extension or the more recent bootstrapped type?