Question about output layer shape / optimal autoencoder architecture

[tb438]

Hi Walter,

Thanks for making your research accomplishments available, compared to other repos your AAE implementation is very easy to understand and work with.

I have two issues I am wondering about, potentially a bug that I can't yet nail down and the other related to your opinion about autoencoder architecture.

I have a collection of IoT devices that I am exploring different options for in terms of generative models (AAE, VAE, and GAN). The goal is to periodically sample an IoT device API to capture state information about various sensors that a human operator has been interacting with, for purposes of creating a generative model latent space that can be sampled from to determine if the current state of the machine is within bounds from an operator perspective.

The dataset I have is in CSV format, all integer valued columns (some of which are int64 in size). So far I have experimented with your unsupervised Wasserstein model, and have swapped out your MNIST functions with functions to vectorize this larger dataset into a float32 feature space.
(the intent was to do everything with float64 but it would appear that the keras normalization layers only support float32).

Here is a weird error I am running across and can't figure out why the output shapes are not matching up:

Traceback (most recent call last):
  File "/home/aj/ga.py", line 315, in <module>
    ae_loss, dc_loss, dc_acc, gen_loss = train_step(batch_x)
  File "/usr/local/lib/python3.7/dist-packages/tensorflow_core/python/eager/def_function.py", line 457, in __call__
    result = self._call(*args, **kwds)
  File "/usr/local/lib/python3.7/dist-packages/tensorflow_core/python/eager/def_function.py", line 503, in _call
    self._initialize(args, kwds, add_initializers_to=initializer_map)
  File "/usr/local/lib/python3.7/dist-packages/tensorflow_core/python/eager/def_function.py", line 408, in _initialize
    *args, **kwds))
  File "/usr/local/lib/python3.7/dist-packages/tensorflow_core/python/eager/function.py", line 1848, in _get_concrete_function_internal_garbage_collected
    graph_function, _, _ = self._maybe_define_function(args, kwargs)
  File "/usr/local/lib/python3.7/dist-packages/tensorflow_core/python/eager/function.py", line 2150, in _maybe_define_function
    graph_function = self._create_graph_function(args, kwargs)
  File "/usr/local/lib/python3.7/dist-packages/tensorflow_core/python/eager/function.py", line 2041, in _create_graph_function
    capture_by_value=self._capture_by_value),
  File "/usr/local/lib/python3.7/dist-packages/tensorflow_core/python/framework/func_graph.py", line 915, in func_graph_from_py_func
    func_outputs = python_func(*func_args, **func_kwargs)
  File "/usr/local/lib/python3.7/dist-packages/tensorflow_core/python/eager/def_function.py", line 358, in wrapped_fn
    return weak_wrapped_fn().__wrapped__(*args, **kwds)
  File "/usr/local/lib/python3.7/dist-packages/tensorflow_core/python/framework/func_graph.py", line 905, in wrapper
    raise e.ag_error_metadata.to_exception(e)
ValueError: in converted code:

    /home/aj/ga.py:243 train_step  *
        ae_loss = autoencoder_loss(batch_x, decoder_output, ae_loss_weight)
    /home/aj/ga.py:204 autoencoder_loss  *
        return loss_weight * mse(inputs, reconstruction)
    /usr/local/lib/python3.7/dist-packages/tensorflow_core/python/keras/losses.py:126 __call__
        losses = self.call(y_true, y_pred)
    /usr/local/lib/python3.7/dist-packages/tensorflow_core/python/keras/losses.py:221 call
        return self.fn(y_true, y_pred, **self._fn_kwargs)
    /usr/local/lib/python3.7/dist-packages/tensorflow_core/python/keras/losses.py:771 mean_squared_error
        return K.mean(math_ops.squared_difference(y_pred, y_true), axis=-1)
    /usr/local/lib/python3.7/dist-packages/tensorflow_core/python/ops/gen_math_ops.py:11014 squared_difference
        "SquaredDifference", x=x, y=y, name=name)
    /usr/local/lib/python3.7/dist-packages/tensorflow_core/python/framework/op_def_library.py:793 _apply_op_helper
        op_def=op_def)
    /usr/local/lib/python3.7/dist-packages/tensorflow_core/python/framework/func_graph.py:548 create_op
        compute_device)
    /usr/local/lib/python3.7/dist-packages/tensorflow_core/python/framework/ops.py:3429 _create_op_internal
        op_def=op_def)
    /usr/local/lib/python3.7/dist-packages/tensorflow_core/python/framework/ops.py:1773 __init__
        control_input_ops)
    /usr/local/lib/python3.7/dist-packages/tensorflow_core/python/framework/ops.py:1613 _create_c_op
        raise ValueError(str(e))

    ValueError: Dimensions must be equal, but are 220 and 223 for 'mean_squared_error/SquaredDifference' (op: 'SquaredDifference') with input shapes: [256,220,28,2], [256,223,28,1].

So after modifying the input shape of your make_encoder_model function, I am getting a 7, 1, 2 output shape instead of the 1, 1, 2 shape of the original model; and, I can't figure out why the make_decoder_outputs function is outputting a 220, 28, 1 shape?

make_encoder_model inputs:  Tensor("input_1:0", shape=(None, 223, 28, 1), dtype=float32)
make_encoder_model outputs:  Tensor("conv2d_4/Identity:0", shape=(None, 7, 1, 2), dtype=float32)
make_decoder_model inputs:  Tensor("input_2:0", shape=(None, 7, 1, 2), dtype=float32)
make_decoder_model outputs:  Tensor("conv2d_10/Identity:0", shape=(None, 220, 28, 2), dtype=float32)
make_discriminator_model inputs:  Tensor("input_3:0", shape=(None, 2), dtype=float32)
make_discriminator_model outputs:  Tensor("dense_2/Identity:0", shape=(None, 1), dtype=float32)

Second question is, what is your opinion about using Conv2D + batch normalization layers for feature detection within this context? Obviously the Conv2D layer is intended for convolution over imagery data; do you have any opinions about the best autoencoder model for a project such as this where the dataset is not imagery data? Or in other words, is Conv2D a good approach to divining features from an arbitrary (non image-based dataset) or would a different keras layer type do a better job of it?

Thanks again for making your research available!