How to disable MXNET_CUDNN_AUTOTUNE_DEFAULT and bucketing log message without turning off MXNET_CUDNN_AUTOTUNE_DEFAULT? #8132

Activity

1. yxchng

   changed the title [-]How to disable MXNET_CUDNN_AUTOTUNE_DEFAULT log message without turning off MXNET_CUDNN_AUTOTUNE_DEFAULT?[/-] [+]How to disable MXNET_CUDNN_AUTOTUNE_DEFAULT and bucketing log message without turning off MXNET_CUDNN_AUTOTUNE_DEFAULT?[/+] on Oct 3, 2017
2. yxchng commented on Oct 3, 2017

   

   yxchng

   on Oct 3, 2017

   Author

   More actions

   I tried exporting or using os.environ but they don't work. The code i am using is https://github.com/pangyupo/mxnet_mtcnn_face_detection
3. szha commented on Oct 5, 2017

   

   szha

   on Oct 5, 2017

   Member

   More actions

   #7988 ought to do it.
4. szha

   added

   Bug

   Operator

   on Oct 5, 2017
5. yxchng commented on Oct 5, 2017

   

   yxchng

   on Oct 5, 2017

   Author

   More actions

   @szha I don't quite understand how to use #7988. How should I solve the problem?
6. szha commented on Oct 5, 2017

   

   szha

   on Oct 5, 2017

   Member

   More actions

   Sorry for the confusion. What I meant was that once that PR is merged the problem should be resolved. Let's wait for that PR to get in first.
7. stupidZZ commented on Oct 13, 2017

   

   stupidZZ

   on Oct 13, 2017

   More actions

   I have the same problems for that.
8. kli-casia commented on Dec 5, 2017

   

   kli-casia

   on Dec 5, 2017

   Contributor

   More actions

   @yxchng did you save this problem, i am also using mtcnn mxnet
9. guker commented on Dec 11, 2017

   

   guker

   on Dec 11, 2017

   More actions

   export MXNET_CUDNN_AUTOTUNE_DEFALUT=0, it is ok
10. jrzaurin commented on Dec 29, 2017

    

    jrzaurin

    on Dec 29, 2017

    Last edited by jrzaurin

    More actions

    Hi there

    I am on Ubuntu 16.04.3 using conda 4.4.2 and Python 2.7.14

    I see the same error even after export MXNET_CUDNN_AUTOTUNE_DEFALUT=0

    ubuntu@ip-10-0-0-14:$ export MXNET_CUDNN_AUTOTUNE_DEFALUT=0
    ubuntu@ip-10-0-0-14:$ python fine_tune_vgg.py --vgg nets/vgg16 --checkpoints checkpoints --prefix vggnet
    [20:56:31] src/io/iter_image_recordio_2.cc:153: ImageRecordIOParser2: ../datasets/rec/train.rec, use 1 threads for decoding..
    [20:56:36] src/io/iter_image_recordio_2.cc:153: ImageRecordIOParser2: ../datasets/rec/val.rec, use 1 threads for decoding..
    [INFO] loading pre-trained model...
    [20:56:37] src/nnvm/legacy_json_util.cc:190: Loading symbol saved by previous version v0.8.0. Attempting to upgrade...
    [20:56:37] src/nnvm/legacy_json_util.cc:198: Symbol successfully upgraded!
    [INFO] training network...
    [20:56:37] src/operator/././cudnn_algoreg-inl.h:112: Running performance tests to find the best convolution algorithm, this can take a while... (setting env variable MXNET_CUDNN_AUTOTUNE_DEFAULT to 0 to disable)
    

    Any news on how to solve this?
11. szha commented on Dec 29, 2017

    

    szha

    on Dec 29, 2017

    Member

    More actions

    @jrzaurin This bug should have already been fixed by the PR mentioned above. Which version of mxnet are you using? Would you be able to upgrade to 1.0?
12. jrzaurin commented on Dec 29, 2017

    

    jrzaurin

    on Dec 29, 2017

    More actions

    @szha

    In fact I am using

    In [2]: mxnet.__version__
    Out[2]: '0.11.0'
    

    which I installed via:
    pip install mxnet-cu80==0.11.0

    (opencv and graphviz where already installed)

    I upgraded to cuDNN v6.0 but I doubt that has something to do (?)

    Overall: Ubuntu 16.04.3, conda 4.4.2, Python 2.7.14 and mxnet 0.11 and I see the same message (I am running it as I write and is there) and it runs very very slow on a tiny dataset (Stanford Cars Dataset)

    so...any idea is appreciated, Thanks!
13. szha commented on Dec 29, 2017

    

    szha

    on Dec 29, 2017

    Member

    More actions

    The latest version of mxnet is mxnet-cu80==1.0.0.post2. Would you be able to try out this version?

    👍React with 👍1Reacted by Javier
14. jrzaurin commented on Dec 30, 2017

    

    jrzaurin

    on Dec 30, 2017

    Last edited by jrzaurin

    More actions

    @szha

    Hey thanks!

    I did not know about the mxnet-cu80==1.0.0, updated on my p2 instance (Tesla K80) and my mac and the message is gone, so I assume that problem solved 👍

    Still runs very slow, but I am sure this is something on my side...so, regarding to the MXNET_CUDNN_AUTOTUNE_DEFALUT message, the solution was to upgrade:

    pip install mxnet-cu80==1.0.0

    and the message disappeared

    thanks!

    EDIT
    @szha

    just updated to mxnet-cu80==1.0.1 and:

    train_net.py:92: DeprecationWarning: mxnet.model.FeedForward has been deprecated. Please use mxnet.mod.Module instead.
      begin_epoch=10)
    /home/ubuntu/anaconda2/lib/python2.7/site-packages/mxnet/model.py:573: DeprecationWarning: Calling initializer with init(str, NDArray) has been deprecated.please use init(mx.init.InitDesc(...), NDArray) instead.
      self.initializer(k, v)
    /home/ubuntu/anaconda2/lib/python2.7/site-packages/mxnet/model.py:579: DeprecationWarning: Calling initializer with init(str, NDArray) has been deprecated.please use init(mx.init.InitDesc(...), NDArray) instead.
      self.initializer(k, v)
    [16:21:55] src/operator/nn/./cudnn/./cudnn_algoreg-inl.h:107: Running performance tests to find the best convolution algorithm, this can take a while... (setting env variable MXNET_CUDNN_AUTOTUNE_DEFAULT to 0 to disable)
    

    The first 3 lines are just DeprecationWarning, but the final one is again the MXNET_CUDNN_AUTOTUNE_DEFAULT message even though is set to 0. Anyway, All seems to work fine, and it runs in decent times.

    I am just saying in case is useful.
15. lanking520 commented on Apr 2, 2018

    

    lanking520

    on Apr 2, 2018

    Member

    More actions

    @yxchng , is this still an issue for you?
16. yxchng commented on Apr 3, 2018

    

    yxchng

    on Apr 3, 2018

    Author

    More actions

    @lanking520 nope thanks
17. yxchng

    closed this as completedon Apr 3, 2018
18. canpeng78 commented on Apr 14, 2018

    

    canpeng78

    on Apr 14, 2018

    More actions

    I'm using mxnet 1.2.0, cuda 9.0 and cudnn 7.0, and still encounter this problem. So what's the final solution? (resetting MXNET_CUDNN_AUTOTUNE_DEFAULT doesn't work).
    Thanks!

    👍React with 👍3Reacted by changya, Manideep and valyriamu
19. arisliang commented on Apr 18, 2018

    

    arisliang

    on Apr 18, 2018

    Last edited by arisliang

    More actions

    While auto tuning, it gives an error message and still running. Is this normal? mx 1.2.0, cuda 9.0, ubuntu 16.04

    2605:2644:0418/091830.114475:ERROR:upload_data_presenter.cc(73)] Not implemented reached in virtual void extensions::RawDataPresenter::FeedNext(const net::UploadElementReader &)

    After export MXNET_CUDNN_AUTOTUNE_DEFAULT = 0, this error was gone.
20. HX1079299660 commented on Nov 16, 2018

    

    HX1079299660

    on Nov 16, 2018

    More actions

    I have the same problems for now
21. sbharadwajj commented on Feb 5, 2019

    

    sbharadwajj

    on Feb 5, 2019

    More actions

    The problem still persists after export MXNET_CUDNN_AUTOTUNE_DEFAULT = 0, running on mxnet 1.3.1, cuda 9.0, ubuntu 18.
    Help would be really appreciated.

    👍React with 👍6Reacted by songwsx, 何飞, Edwardmark, Eduardo, Etpoem and Daniel (Neng) Wang
22. szha

    reopened this on Feb 5, 2019
23. apeforest commented on Feb 21, 2019

    

    apeforest

    on Feb 21, 2019

    Contributor

    More actions

    Can you check in your model, if some convolution layer has cudnn_tune not set to off. If so, please update your model by setting all the cudnn_tune param all the layers to off. Thanks.

    👍React with 👍4Reacted by Eduardo, Flavien Berwick, Ryo Wakabayashi and Armen Ghambaryan

    🎉React with 🎉3Reacted by Eduardo, Ryo Wakabayashi and maoxian
24. joanshen0508 commented on Aug 26, 2021

    

    joanshen0508

    on Aug 26, 2021

    More actions

    1. you should revise your model file (*.json). Modify "cudnn_tune": "limited_workspace", to "cudnn_tune": "None",
    2. add
       import os
       os.environ['MXNET_CUDNN_AUTOTUNE_DEFAULT'] = '0'
       it works for me.
25. apache

    locked and limited conversation to collaborators on Aug 26, 2021
26. szha

    closed this as completedon Aug 26, 2021