Pre-trained models - Error after changing "trainable" from 0 to 50

I got an error when I started the training after changing “trainable” from 0 to 50
The first run with Trainable set to zero started Ok and ran OK.

Actually the second run (trainable set to 50) was restarted from the first run where trainable is set to 0.

Any news on this topic?

We are able to reproduce this issue. It seems to always happen when a pre-trained model is used. The issue seems to be coming because of MxNet backend.

We need to investigate further in MxNet backend to make a fix or move to newer version with Keras 2.

Thanks for the information.

This feature can be very useful for transfert learning when, in a first step, one tries to just retrain the fully connected layer taking the pre-trained network as it is (with ImageNet weights) and then, in a second step one tries to fine-tune the last few layer on the pre-trained model.

When is the next version planned ? Will it be updated to Keras 2 ?

Next version is 2.1.0. Expected to be released in next few days. It has performance improvement for training, inference and some of the suggestions mentioned in this forum. It does not have keras 2.

Thanks for the information. Great that the inference is faster because right now it has a strong impact on the DLS training performance (when using a validation set). It will be calculated on the GPU ?

Inference is still on the CPU but performance is improved by using all the CPU threads. So you could see 12 times performance improvement (on i7 CPU with 6 cores) .

Thanks Rajendra,

Is it due to the MXNet framework that inference can’t be calculated on the GPU ? or is it another issue like communication overload, memory management, … ?

Yes, it is because of MxNet backend. By default the Keras predict function does not use the GPU on MxNet backend. I have found a thread where user is able to get it to run with a workaround. We need to check and verify that this works.

Ok so more a Keras 1.2.2 issue.

The fix could be worth a try. Otherwise maybe Keras 2 will work Ok with MXNet and do the inferring on the GPU by default.

We have tried the fix and it works. We will include it in the upcoming release.

1 Like

Great news. :smile: hanks for the effort.