Is there a way to use H.264 with custom color repersentation?

Question

I'm currently developing new file format, it's a video with custom color representation. Every color is a single byte; there's a constant RGBA colors array, every byte of the frame is the index of color in this array. Therefore every pixel in a single byte.

So I'm looking for a way to compress videos with such format. My first idea was is create this video format myself (which unfortunately failed), second idea is H.264, but I don't know if there's any way to use H.264 this way. So is there? Or maybe there's another way to compress such video data? (except gzip, lzma, bzip2, 7zip and so on)

Please, don't close this question. I'll add all asked details if needed.

Answer 1

The best I can suggest for such idea would be to encode it in 4:0:0 (mono) colorspace in lossless mode in H.264. For x264 this would mean options: --input-csp i400 --output-csp i400 --qp 0. But I doubt motion compensation would be good in such palette colorspace.