I'm new to deep learning so if the question doesn't make sense plz correct me.
In traditional machine learning I know how to compare models and choose one of the as the best with the metrics I choose.
However, in deep learning, each model is build with different layers, so how can I control variables to determine which model is the best fairly? Or usually people don't compare in this way?
For example I have a sequential data, I can use both CNN and LSTM model, so should I compare model with only one layer of CNN and one layer of LSTM? After that I can add more layers or tuning my model?
Or someone can just tell me the process of how to compare and choose the best deep learning model with chosen metrics?