Problem with inference on model with 0.43 validation accuracy

I have trained a DL model and I get the following performance:


During the training the “Save weight on” param was set to “Best accuracy”.
So from the validation accuracy graph, one can see that the best value was 0.43.
When doing the inferencing:

only 9 images out of 1022 are classified properly. That’s less than 1% accuracy. Looks like there’s a problem.

Any news on this issue ?

Sorry for the delay in getting back on this. I was able to reproduce this but it needed further investigation to find the root cause.

This is another MXNet backend bug happening when pre-trained model is used. There is one issue filed on github.

Can you try inference from Jupyter notebook and theano backend?

Thanks for investigation. In a way, it is reassuring. Which version of MXNet is used in DLS?

Ok I will try inferencing from Jupyter Notebook.

keras-mxnet=1.2.2.1 and mxnet==0.12.1

Looks like Keras-MXNet just added support for Keras 2:


this is very unfortunate :frowning: I think lot of people will try to copy your example videon (hotel room stars) in the soda competiton, and they will get a bad experience. Can you add some warning on the inference page about this issue?!
Also it would be good to update the video with the preprocessing and resize settings, and make clear to use the normalization and downscaling settings.

Hi @Deep_In_Depth,

Thanks for bringing this to our attention, this bug has been fixed and will be released shortly.
Happy to hear from you!

Thanks for the concern.
The issue is fixed and will be released soon, the soda bottle competition will not have any effect. :slight_smile:

About the video, will consider your feedback. Thanks again!

2 Likes

Great news! Thank you!