astropy.io.fits - HIERARCH keywords don't work with CONTINUE cards: Bug or "Feature" of the FITS standard?

Question

The astropy.io.fits manual states, that we can use header keywords longer than 8-characters. In this case HIERARCH cards will be created. The manual also states, that if we want to store keyword-value pairs longer than 80-characters, continue cards will automatically be created.

However, in practice it seems that both definitions work only mutually exclusive, i.e. we can not create a FITS file containing a keyword value pair, where the keyword is longer than 8-characters (i.e. a HIERARCH keyword) and the value is a very long string.

An example:

from astropy.io import fits

header1 = fits.Header()
header2 = fits.Header()
header3 = fits.Header()

header1['TEST'] = 'superlongstring'*10
header2['TEST TEST'] = 'superlongstring'
header3['TEST TEST'] = 'superlongstring'*10

Here header1 and header2 will be correct, but when calling repr(header3) or attempting to save a FITS file with such a header, the error ValueError: The keyword TEST TEST with its value is too long is raised.

Is this an "unintended feature" of the FITS standard, i.e. can HIERARCH keywords not be continued with CONTINUE cards or might this be simply a bug of astropy.io.fits?

Answer 1

The following part of the answer is courtesy of a co-worker, who is familiar with the astropy codebase.

Looking at the astropy source code, it is explicitly ruled out to use CONTINUE cards in a value belonging to a HIERARCH keyword.

In the source code we find (astropy/io/fits/card.py#L1227-1236):

keywordvalue_length = len(keyword) + len(delimiter) + len(value)
if (keywordvalue_length > self.length and
        keyword.startswith('HIERARCH')):
    if (keywordvalue_length == self.length + 1 and keyword[-1] == ' '):
        output = ''.join([keyword[:-1], delimiter, value, comment])
    else:
        # I guess the HIERARCH card spec is incompatible with CONTINUE
        # cards
        raise ValueError('The keyword %s with its value is too long' %
                         self.keyword)

The comment is from 2011, and was put there during a redesign of pyfits. Before that, pyfits could also read or write exlusively either HIERARCH cards or cards with CONTINUE statement, and apparently it was kept like that. The relevant old PyFITS code is:

if cardimage[:8].upper() == 'HIERARCH':
    card = _HierarchCard()
    # for card image longer than 80, assume it contains CONTINUE card(s).
elif len(cardimage) > Card.length:
    card = _ContinueCard()

However, according to my feeling there is no special reason for HIERARCH excluding long values with CONTINUE.

So far the answer from my co-worker, who closes by recommending the creation of an ticket in the astropy issue tracker.

I also did some research myself. Both, the HIERARCH and the CONTINUE keywords are not part of the official FITS standard (Pence, W.D. et al. 2010, Astronomy & Astrophysics Vol. 524, A42). The HIERARCH keyword convention is given in Wiecencec, A. et al. 2009, "The ESO HIERARCH Keyword Convention") and the CONTINUE convention is given by the HEASARC FITS Working Group, 2007. "The CONTINUE Long String Keyword Convention". Reading carefully both of these definitions, I see absolutely no reason why they should be mutually exclusive. Hence, I created an issue in the astropy issue tracker.

Edit: As mentioned in the answer by Iguananaut - there appears to be one reason why they are indeed mutually exclusive, namely that HIERARCH cards formally do not contain any value at all... Crazy but true and therefore I think my answer is not correct.

Answer 2

I wrote this in the issue opened by the OP, but I'm copying a version of it here too as a possible answer:

The issue here is that the CONTINUE convention is strictly limited to extending the value of cards with string values (i.e. it does not apply to cards with different types of values). However, formally speaking, cards using the HIERARCH convention do not have any value at all, and so the CONTINUE convention does not apply to it, unless one were to implicitly support the CONTINUE convention within parsing of cards using the HIERARCH convention. Because there are ambiguities in how to do this (for example what is the maximum length of a keyword?) and because the HIERARCH convention does not explicitly allow for this interpretation (it gives a specific set of guidelines for how to interpret the contents of a HIERARCH card) FITS readers should refrain from adding any implicit support for some undocumented interpretation of HIERARCH cards (thus insituting yet another implicit convention that would not be supported by most readers/writers).

What one could do is lobby the authors of the ESO HIERARCH convention to update their convention for explicit mutual compatibility with the CONTINUE convention, at which time this should be allowed in any FITS readers that support both conventions. But in the meantime the issue is too fraught with difficulties, even if it seems "obvious" that they could work together.