When switching from federated to centralized dataset -Error may indicate that you're trying to pass a Tensor to a NumPy call

Question

I'm trying to switch from a federated setting to centralized learning. I've created a federated dataset, but I want to create a dataset for centralized learning with the create_tf_dataset_from_all_clients function. When I googled the error I found out that maybe versions of NumPy and TensorFlow are not correct for this function, my current versions are :

• python == 3.9
• tensorflow==2.8.2
• numpy==1.21.6
• tensorflow-federated==0.24.0

I haven't found some recent posts about TensorFlow 2.8 and matching NumPy version

Also, the error might come from a function that I used to create the clientData object:

 def parse_image(filename):
    parts = tf.strings.split(filename, os.sep)
    label_str = parts[-2]

    label_int = tf.where(labels_tf == label_str)[0][0]

    image = tf.io.read_file(filename)
    image = tf.io.decode_jpeg(image, channels=3)
    image = tf.image.convert_image_dtype(image, tf.float32)
    image = tf.image.resize(image, [32, 32])
    image = tf.keras.applications.resnet50.preprocess_input(image)

    if base_model == "VGG16":
        print("-------- preprocessing image for base_model VGG16 --------")

        image = tf.keras.applications.vgg16.preprocess_input(image)

    elif base_model == "ResNet":
        print("-------- preprocessing image for base_model  ResNet --------")

        image = tf.keras.applications.resnet.preprocess_input(image)

    return image, label_int

def create_dataset(client_id):

    df = train_set

    client_id = int(client_id)

    file = df.loc[df["client_id"] == client_id]
    # print(file)
    path = file["path"]

    # print(path)
    list_ds = tf.data.Dataset.list_files(path)

    images_ds = list_ds.map(parse_image)

    return images_ds

Error:

TypeError                                 Traceback (most recent call last)
Input In [7], in <cell line: 1>()
----> 1 train_dataset = client_data.create_tf_dataset_from_all_clients()

File ~/master_venv/lib/python3.9/site-packages/tensorflow_federated/python/simulation/datasets/client_data.py:231, in ClientData.create_tf_dataset_from_all_clients(self, seed)
    227 nested_dataset = tf.data.Dataset.from_tensor_slices(client_ids)
    228 # We apply serializable_dataset_fn here to avoid loading all client datasets
    229 # in memory, which is slow. Note that tf.data.Dataset.map implicitly wraps
    230 # the input mapping in a tf.function.
--> 231 example_dataset = nested_dataset.flat_map(self.serializable_dataset_fn)
    232 return example_dataset

File ~/master_venv/lib/python3.9/site-packages/tensorflow/python/data/ops/dataset_ops.py:2092, in DatasetV2.flat_map(self, map_func, name)
   2058 def flat_map(self, map_func, name=None):
   2059   """Maps `map_func` across this dataset and flattens the result.
   2060 
   2061   The type signature is:
   (...)
   2090     Dataset: A `Dataset`.
   2091   """
-> 2092   return FlatMapDataset(self, map_func, name=name)

File ~/master_venv/lib/python3.9/site-packages/tensorflow/python/data/ops/dataset_ops.py:5327, in FlatMapDataset.__init__(self, input_dataset, map_func, name)
   5325 """See `Dataset.flat_map()` for details."""
   5326 self._input_dataset = input_dataset
-> 5327 self._map_func = structured_function.StructuredFunctionWrapper(
   5328     map_func, self._transformation_name(), dataset=input_dataset)
   5329 if not isinstance(self._map_func.output_structure, DatasetSpec):
   5330   raise TypeError(
   5331       "The `map_func` argument must return a `Dataset` object. Got "
   5332       f"{_get_type(self._map_func.output_structure)!r}.")

File ~/master_venv/lib/python3.9/site-packages/tensorflow/python/data/ops/structured_function.py:271, in StructuredFunctionWrapper.__init__(self, func, transformation_name, dataset, input_classes, input_shapes, input_types, input_structure, add_to_graph, use_legacy_function, defun_kwargs)
    264       warnings.warn(
    265           "Even though the `tf.config.experimental_run_functions_eagerly` "
    266           "option is set, this option does not apply to tf.data functions. "
    267           "To force eager execution of tf.data functions, please use "
    268           "`tf.data.experimental.enable_debug_mode()`.")
    269     fn_factory = trace_tf_function(defun_kwargs)
--> 271 self._function = fn_factory()
    272 # There is no graph to add in eager mode.
    273 add_to_graph &= not context.executing_eagerly()

File ~/master_venv/lib/python3.9/site-packages/tensorflow/python/eager/function.py:2567, in Function.get_concrete_function(self, *args, **kwargs)
   2558 def get_concrete_function(self, *args, **kwargs):
   2559   """Returns a `ConcreteFunction` specialized to inputs and execution context.
   2560 
   2561   Args:
   (...)
   2565        or `tf.Tensor` or `tf.TensorSpec`.
   2566   """
-> 2567   graph_function = self._get_concrete_function_garbage_collected(
   2568       *args, **kwargs)
   2569   graph_function._garbage_collector.release()  # pylint: disable=protected-access
   2570   return graph_function

File ~/master_venv/lib/python3.9/site-packages/tensorflow/python/eager/function.py:2533, in Function._get_concrete_function_garbage_collected(self, *args, **kwargs)
   2531   args, kwargs = None, None
   2532 with self._lock:
-> 2533   graph_function, _ = self._maybe_define_function(args, kwargs)
   2534   seen_names = set()
   2535   captured = object_identity.ObjectIdentitySet(
   2536       graph_function.graph.internal_captures)

File ~/master_venv/lib/python3.9/site-packages/tensorflow/python/eager/function.py:2711, in Function._maybe_define_function(self, args, kwargs)
   2708   cache_key = self._function_cache.generalize(cache_key)
   2709   (args, kwargs) = cache_key._placeholder_value()  # pylint: disable=protected-access
-> 2711 graph_function = self._create_graph_function(args, kwargs)
   2712 self._function_cache.add(cache_key, cache_key_deletion_observer,
   2713                          graph_function)
   2715 return graph_function, filtered_flat_args

File ~/master_venv/lib/python3.9/site-packages/tensorflow/python/eager/function.py:2627, in Function._create_graph_function(self, args, kwargs)
   2622 missing_arg_names = [
   2623     "%s_%d" % (arg, i) for i, arg in enumerate(missing_arg_names)
   2624 ]
   2625 arg_names = base_arg_names + missing_arg_names
   2626 graph_function = ConcreteFunction(
-> 2627     func_graph_module.func_graph_from_py_func(
   2628         self._name,
   2629         self._python_function,
   2630         args,
   2631         kwargs,
   2632         self.input_signature,
   2633         autograph=self._autograph,
   2634         autograph_options=self._autograph_options,
   2635         arg_names=arg_names,
   2636         capture_by_value=self._capture_by_value),
   2637     self._function_attributes,
   2638     spec=self.function_spec,
   2639     # Tell the ConcreteFunction to clean up its graph once it goes out of
   2640     # scope. This is not the default behavior since it gets used in some
   2641     # places (like Keras) where the FuncGraph lives longer than the
   2642     # ConcreteFunction.
   2643     shared_func_graph=False)
   2644 return graph_function

File ~/master_venv/lib/python3.9/site-packages/tensorflow/python/framework/func_graph.py:1141, in func_graph_from_py_func(name, python_func, args, kwargs, signature, func_graph, autograph, autograph_options, add_control_dependencies, arg_names, op_return_value, collections, capture_by_value, acd_record_initial_resource_uses)
   1138 else:
   1139   _, original_func = tf_decorator.unwrap(python_func)
-> 1141 func_outputs = python_func(*func_args, **func_kwargs)
   1143 # invariant: `func_outputs` contains only Tensors, CompositeTensors,
   1144 # TensorArrays and `None`s.
   1145 func_outputs = nest.map_structure(
   1146     convert, func_outputs, expand_composites=True)

File ~/master_venv/lib/python3.9/site-packages/tensorflow/python/data/ops/structured_function.py:248, in StructuredFunctionWrapper.__init__.<locals>.trace_tf_function.<locals>.wrapped_fn(*args)
    242 @eager_function.defun_with_attributes(
    243     input_signature=structure.get_flat_tensor_specs(
    244         self._input_structure),
    245     autograph=False,
    246     attributes=defun_kwargs)
    247 def wrapped_fn(*args):  # pylint: disable=missing-docstring
--> 248   ret = wrapper_helper(*args)
    249   ret = structure.to_tensor_list(self._output_structure, ret)
    250   return [ops.convert_to_tensor(t) for t in ret]

File ~/master_venv/lib/python3.9/site-packages/tensorflow/python/data/ops/structured_function.py:177, in StructuredFunctionWrapper.__init__.<locals>.wrapper_helper(*args)
    175 if not _should_unpack(nested_args):
    176   nested_args = (nested_args,)
--> 177 ret = autograph.tf_convert(self._func, ag_ctx)(*nested_args)
    178 if _should_pack(ret):
    179   ret = tuple(ret)

File ~/master_venv/lib/python3.9/site-packages/tensorflow/python/autograph/impl/api.py:692, in convert.<locals>.decorator.<locals>.wrapper(*args, **kwargs)
    690 except Exception as e:  # pylint:disable=broad-except
    691   if hasattr(e, 'ag_error_metadata'):
--> 692     raise e.ag_error_metadata.to_exception(e)
    693   else:
    694     raise

File ~/master_venv/lib/python3.9/site-packages/tensorflow/python/autograph/impl/api.py:689, in convert.<locals>.decorator.<locals>.wrapper(*args, **kwargs)
    687 try:
    688   with conversion_ctx:
--> 689     return converted_call(f, args, kwargs, options=options)
    690 except Exception as e:  # pylint:disable=broad-except
    691   if hasattr(e, 'ag_error_metadata'):

File ~/master_venv/lib/python3.9/site-packages/tensorflow/python/autograph/impl/api.py:439, in converted_call(f, args, kwargs, caller_fn_scope, options)
    437 try:
    438   if kwargs is not None:
--> 439     result = converted_f(*effective_args, **kwargs)
    440   else:
    441     result = converted_f(*effective_args)

File /var/folders/w2/fcxhc9j52tb9hymgw1b8_dmh0000gn/T/__autograph_generated_filepc7z792y.py:11, in outer_factory.<locals>.inner_factory.<locals>.tf__create_dataset(client_id)
      9 retval_ = ag__.UndefinedReturnValue()
     10 client_id = ag__.converted_call(ag__.ld(int), (ag__.ld(client_id),), None, fscope)
---> 11 files = ag__.ld(df).loc[ag__.ld(df)['client_id'] == ag__.ld(client_id)]
     12 path = ag__.ld(files)['path']
     13 list_ds = ag__.converted_call(ag__.ld(tf).data.Dataset.list_files, (ag__.ld(path),), None, fscope)

File ~/master_venv/lib/python3.9/site-packages/pandas/core/ops/common.py:70, in _unpack_zerodim_and_defer.<locals>.new_method(self, other)
     66             return NotImplemented
     68 other = item_from_zerodim(other)
---> 70 return method(self, other)

File ~/master_venv/lib/python3.9/site-packages/pandas/core/arraylike.py:40, in OpsMixin.__eq__(self, other)
     38 @unpack_zerodim_and_defer("__eq__")
     39 def __eq__(self, other):
---> 40     return self._cmp_method(other, operator.eq)

File ~/master_venv/lib/python3.9/site-packages/pandas/core/series.py:5625, in Series._cmp_method(self, other, op)
   5622 with np.errstate(all="ignore"):
   5623     res_values = ops.comparison_op(lvalues, rvalues, op)
-> 5625 return self._construct_result(res_values, name=res_name)

File ~/master_venv/lib/python3.9/site-packages/pandas/core/series.py:3017, in Series._construct_result(self, result, name)
   3013     return (res1, res2)
   3015 # We do not pass dtype to ensure that the Series constructor
   3016 #  does inference in the case where `result` has object-dtype.
-> 3017 out = self._constructor(result, index=self.index)
   3018 out = out.__finalize__(self)
   3020 # Set the result's name after __finalize__ is called because __finalize__
   3021 #  would set it back to self.name

File ~/master_venv/lib/python3.9/site-packages/pandas/core/series.py:442, in Series.__init__(self, data, index, dtype, name, copy, fastpath)
    440     index = default_index(len(data))
    441 elif is_list_like(data):
--> 442     com.require_length_match(data, index)
    444 # create/copy the manager
    445 if isinstance(data, (SingleBlockManager, SingleArrayManager)):

File ~/master_venv/lib/python3.9/site-packages/pandas/core/common.py:556, in require_length_match(data, index)
    552 def require_length_match(data, index: Index):
    553     """
    554     Check the length of data matches the length of the index.
    555     """
--> 556     if len(data) != len(index):
    557         raise ValueError(
    558             "Length of values "
    559             f"({len(data)}) "
    560             "does not match length of index "
    561             f"({len(index)})"
    562         )

File ~/master_venv/lib/python3.9/site-packages/tensorflow/python/framework/ops.py:932, in Tensor.__len__(self)
    931 def __len__(self):
--> 932   raise TypeError(f"len is not well defined for a symbolic Tensor "
    933                   f"({self.name}). Please call `x.shape` rather than "
    934                   f"`len(x)` for shape information.")

TypeError: in user code:

    File "/var/folders/w2/fcxhc9j52tb9hymgw1b8_dmh0000gn/T/ipykernel_2264/3413278942.py", line 7, in create_dataset  *
        files = df.loc[df['client_id']==client_id]
    File "/Users/admin/master_venv/lib/python3.9/site-packages/pandas/core/ops/common.py", line 70, in new_method
        return method(self, other)
    File "/Users/admin/master_venv/lib/python3.9/site-packages/pandas/core/arraylike.py", line 40, in __eq__
        return self._cmp_method(other, operator.eq)
    File "/Users/admin/master_venv/lib/python3.9/site-packages/pandas/core/series.py", line 5625, in _cmp_method
        return self._construct_result(res_values, name=res_name)
    File "/Users/admin/master_venv/lib/python3.9/site-packages/pandas/core/series.py", line 3017, in _construct_result
        out = self._constructor(result, index=self.index)
    File "/Users/admin/master_venv/lib/python3.9/site-packages/pandas/core/series.py", line 442, in __init__
        com.require_length_match(data, index)
    File "/Users/admin/master_venv/lib/python3.9/site-packages/pandas/core/common.py", line 556, in require_length_match
        if len(data) != len(index):

    TypeError: len is not well defined for a symbolic Tensor (Equal:0). Please call `x.shape` rather than `len(x)` for shape information.

Answer 1

TFF is generally programmed assuming that 'all' local logic is expressed in pure TensorFlow (or at least, can be hoisted into a platform-independent representation, like TorchScript, GraphDef or XLA); this is crucial for TFF's "write-once, run-everywhere" philosophy, to prevent capturing of arbitrary python code.

It is this assumption that is surfacing here. TFF passes your function directly to TensorFlow libraries which implicitly create a tf.function; you can in fact see this in the stacktrace above:

    228 # We apply serializable_dataset_fn here to avoid loading all client datasets
    229 # in memory, which is slow. Note that tf.data.Dataset.map implicitly wraps
    230 # the input mapping in a tf.function.
--> 231 example_dataset = nested_dataset.flat_map(self.serializable_dataset_fn)

While TF is trying to create this tf.function, or invoke it, it will pass a tensor through the function and attempt to trace the logic. This makes working with Python datastructures a little difficult; e.g., we cannot use this tensor to index into a list or dict. However, it looks to me like if you construct a tf.lookup.StaticHashTable with your clients-to-files mapping, and look up with your client ID in this hash table instead of the pandas dataframe, your code may 'just work'.

In general, you can test whether your code will work or not with this usage of TFF and tf.data by wrapping the function you pass to TFF as a tf.function, invoking it, and making sure it has the behavior you expect.