9

What is the best process for making a required field optional in thrift. For example, I have a struct...

struct Message {
    1: required double userID;
    2: required string content;
    ...
}

... but I want to make content optional.

EDIT: To clarify, I already have consumers that use this struct, so I would need to update it without breaking those consumers. A staged upgrade is fine (ie - add a new optional field, update the downstream clients, then remove--or stop using--the old required field).

JensG
  • 13,148
  • 4
  • 45
  • 55
AJH
  • 135
  • 1
  • 1
  • 4

2 Answers2

16

You can't, because as the saying goes required is forever. The following is a quote from Diwaker Gupta's highly recommendable "Missing Guide". It pretty much nails it down why one should think twice (at least) before using required:

Required Is Forever

You should be very careful about marking fields as required. If at some point you wish to stop writing or sending a required field, it will be problematic to change the field to an optional field — old readers will consider messages without this field to be incomplete and may reject or drop them unintentionally. You should consider writing application-specific custom validation routines for your buffers instead. Some have come the conclusion that using required does more harm than good; they prefer to use only optional. However, this view is not universal.

I'm afraid that the only option is to deprecate the entire struct and create a new one.

In addition to that, there are in fact three degrees of requiredness, only two of them have keywords:

  • required: must exist on read, must be set on write
  • optional: may or may not be set, entirely optional
  • "default": may not exist on read, always written (unless it is a null pointer )

The "default" requiredness is applied implicitly when neither required nor optional are specified.

As one clearly sees, the restrictions for required are rather strict if we look at the compatibility site of things. Even adding a new required field to a struct may lead to incompatibility, e.g. if a new client is reading data from an old server (or vice versa), because the new required field is not in the data written by the old impl, but expected by the new impl.

JensG
  • 13,148
  • 4
  • 45
  • 55
  • The statement goes towards "readers" but what about old _writers_. If there is only one reader and it changes the contract to optional, but legacy writers are still writing it as required is that going to break? – devshorts Nov 08 '16 at 22:15
  • 1
    "*If there is only one reader and it changes the contract to optional, but legacy writers are still writing it as required is that going to break?*" -- No, that specific constellation should work. It's against recommended practice to modify an published API, but yes, that should work. – JensG Nov 09 '16 at 09:41
2

It is possible to do this if you control all the readers and writers of the struct in the following steps :

  1. Change the field in the thrift schema from required to optional. Sync this new schema in all the systems that refer to or use this struct. Do not stop writing this field. This is a safe change as the field will still be always present.
  2. Only after the schema change has been deployed in all the systems using this struct, you can stop writing this field in some cases. This is now safe to do as this field is marked as optional in every system, there are no "old readers" as referred to in the other answer.
Hitesh Gupta
  • 333
  • 1
  • 3
  • 8