1

I would like to run multiple experiments, then report model accuracy per experiment.

I'm training a toy MNIST example with pytorch (v1.1.0), but the goal is, once I can compare performance for the toy problem, to have it integrated with the actual code base.

As I understand the TRAINS python package, with the "two lines of code" all my hyper-parameters are already logged (Command line argparse in my case).

What do I need to do in order to report a final scalar and then be able to sort through all the different training experiments (w/ hyper-parameters) in order to find the best one.

What I'd like to get, is a graph/s where on the X-axis I have hyper-parameter values and on the Y-axis I have the validation accuracy.

DalyaG
  • 2,979
  • 2
  • 16
  • 19
fidlr
  • 81
  • 5

1 Answers1

1

I assume you are referring to: https://pypi.org/project/trains/ (https://github.com/allegroai/trains), which I'm one of the maintainers.

You can manually create a plot with a single point X-axis for the hyper-parameter value, and Y-Axis for the accuracy. 

number_layers = 10
accuracy = 0.95
Task.current_task().get_logger().report_scatter2d(
    "performance", "accuracy", iteration=0, 
    mode='markers', scatter=[(number_layers, accuracy)])

Assuming your hyper-parameter is "number_layers" with current value 10, and the accuracy for the trained model is 0.95.

Then when you compare the experiments you get something like that:

compare scatter plots

Martin.B
  • 599
  • 3
  • 9
  • You could also have a histogram instead of 2D scatter plot: Task.current_task().get_logger().report_vector( "performance", "accuracy", iteration=0, labels=['accuracy'], values=[accuracy], xlabels=['number_layers %d' % number_layers]) – Martin.B Jun 24 '19 at 22:13
  • great! I like the histogram even better Thanks – fidlr Jun 24 '19 at 22:31
  • @Martin.B it looks like you have some kind of affiliation with the TRAINS project. There is nothing wrong about it. However, [users are expected to disclose their affiliations](https://stackoverflow.com/help/behavior) in these cases. Would you mind to add a disclaimer disclosing it? – Berriel Jun 24 '19 at 22:51
  • @Berriel my apologies, I now made sure my affiliation is noticable in my profile. I will also make sure it is clear on my next replies – Martin.B Jun 26 '19 at 19:37
  • @Martin.B I think you didn't read the link I posted. You must disclose the affiliation on your answers. Doing it on your profile is your choice. Can you update this answer with a disclaimer? – Berriel Jun 26 '19 at 22:26
  • @Berriel, sure I edited the original answer to include my affiliation, let me know if it is clear enough. – Martin.B Jun 27 '19 at 18:17