504 gateway timeout error

After training the model, then I deployed the instance after which it generated a link. when I clicked on the link it kept on showing 504 gateway timeout. Please what do I do?

Hi @Jamiepoo17,

It’s look like your model is not deployed properly. 504 gateway error came when gateway did not get a response in time from the upstream server that it needed in order to complete the request.

Can you please share the logs so that we can help you out?

Thanks,
Sandhaya Kumari

Thanks @sandhaya.choudhary. Here is the log

[1;34mrun_webapp_servere[0m
skipping minc-2500-tiny dataset download!
skipping cifar dataset download!
skipping IMDB dataset download!
skipping MNIST dataset download!
skipping coco-400-person-dog dataset download!
skipping reuters dataset download!
Initializing DLS db… \n
Checking for environments update …
2020-08-03 04:03:27.399518: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library cudart64_101.dll
DISABLE_LOCAL_LOGIN is TRUE
No changes detected
2020-08-03 04:03:36.293449: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library cudart64_101.dll
DISABLE_LOCAL_LOGIN is TRUE
Operations to perform:
Apply all migrations: account, admin, auth, authtoken, automl, contenttypes, environments, project, projects, reversion, sessions, sites, socialaccount
Running migrations:
No migrations to apply.
2020-08-03 04:03:42.175321: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library cudart64_101.dll
DISABLE_LOCAL_LOGIN is TRUE
Installed 2 object(s) from 1 fixture(s)
e[1;34m Starting DLS backend server at 127.0.0.1:22002
e[0m
2020-08-03 04:03:46.112089: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library cudart64_101.dll
2020-08-03 04:03:48.629039: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library nvcuda.dll
2020-08-03 04:03:49.202010: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1555] Found device 0 with properties:
pciBusID: 0000:01:00.0 name: GeForce GTX 960M computeCapability: 5.0
coreClock: 1.176GHz coreCount: 5 deviceMemorySize: 2.00GiB deviceMemoryBandwidth: 74.65GiB/s
2020-08-03 04:03:49.202409: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library cudart64_101.dll
2020-08-03 04:03:49.285584: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library cublas64_10.dll
2020-08-03 04:03:49.366395: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library cufft64_10.dll
2020-08-03 04:03:49.384175: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library curand64_10.dll
2020-08-03 04:03:49.481436: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library cusolver64_10.dll
2020-08-03 04:03:49.530896: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library cusparse64_10.dll
2020-08-03 04:03:49.721027: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library cudnn64_7.dll
2020-08-03 04:03:50.032063: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1697] Adding visible gpu devices: 0
DISABLE_LOCAL_LOGIN is TRUE
[03/Aug/2020:04:03:50] ENGINE Bus STARTING
INFO:cherrypy.error:[03/Aug/2020:04:03:50] ENGINE Bus STARTING
[03/Aug/2020:04:03:51] ENGINE Serving on http://127.0.0.1:22002
INFO:cherrypy.error:[03/Aug/2020:04:03:51] ENGINE Serving on http://127.0.0.1:22002
[03/Aug/2020:04:03:51] ENGINE Bus STARTED
INFO:cherrypy.error:[03/Aug/2020:04:03:51] ENGINE Bus STARTED
{‘Image’: {‘port’: ‘InputPort0’, ‘type’: ‘Image’, ‘categories’: 277524}, ‘Rating’: {‘port’: ‘OutputPort0’, ‘type’: ‘Categorical’, ‘categories’: 2}}
{‘Image’: {‘port’: ‘InputPort0’, ‘type’: ‘Image’, ‘categories’: 277524}, ‘Rating’: {‘port’: ‘OutputPort0’, ‘type’: ‘Categorical’, ‘categories’: 2}}
2020-08-03 04:05:37.129091: I tensorflow/core/platform/cpu_feature_guard.cc:142] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX AVX2
2020-08-03 04:05:37.133225: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1096] Device interconnect StreamExecutor with strength 1 edge matrix:
2020-08-03 04:05:37.133471: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1102]
{‘Image’: {‘port’: ‘InputPort0’, ‘type’: ‘Image’, ‘categories’: 277524}, ‘Rating’: {‘port’: ‘OutputPort0’, ‘type’: ‘Categorical’, ‘categories’: 2}}
{‘Image’: {‘port’: ‘InputPort0’, ‘type’: ‘Image’, ‘categories’: 277524}, ‘Rating’: {‘port’: ‘OutputPort0’, ‘type’: ‘Categorical’, ‘categories’: 2}}
{‘Image’: {‘port’: ‘InputPort0’, ‘type’: ‘Image’, ‘categories’: 277524}, ‘Rating’: {‘port’: ‘OutputPort0’, ‘type’: ‘Categorical’, ‘categories’: 2}}
{‘Image’: {‘port’: ‘InputPort0’, ‘type’: ‘Image’, ‘categories’: 277524}, ‘Rating’: {‘port’: ‘OutputPort0’, ‘type’: ‘Categorical’, ‘categories’: 2}}
{‘Image’: {‘port’: ‘InputPort0’, ‘type’: ‘Image’, ‘categories’: 277524}, ‘Rating’: {‘port’: ‘OutputPort0’, ‘type’: ‘Categorical’, ‘categories’: 2}}
{‘Image’: {‘port’: ‘InputPort0’, ‘type’: ‘Image’, ‘categories’: 277524}, ‘Rating’: {‘port’: ‘OutputPort0’, ‘type’: ‘Categorical’, ‘categories’: 2}}
{‘Image’: {‘port’: ‘InputPort0’, ‘type’: ‘Image’, ‘categories’: 277524}, ‘Rating’: {‘port’: ‘OutputPort0’, ‘type’: ‘Categorical’, ‘categories’: 2}}
{‘Image’: {‘port’: ‘InputPort0’, ‘type’: ‘Image’, ‘categories’: 277524}, ‘Rating’: {‘port’: ‘OutputPort0’, ‘type’: ‘Categorical’, ‘categories’: 2}}
{‘Image’: {‘port’: ‘InputPort0’, ‘type’: ‘Image’, ‘categories’: 277524}, ‘Rating’: {‘port’: ‘OutputPort0’, ‘type’: ‘Categorical’, ‘categories’: 2}}
{‘Image’: {‘port’: ‘InputPort0’, ‘type’: ‘Image’, ‘categories’: 277524}, ‘Rating’: {‘port’: ‘OutputPort0’, ‘type’: ‘Categorical’, ‘categories’: 2}}
{‘Image’: {‘port’: ‘InputPort0’, ‘type’: ‘Image’, ‘categories’: 277524}, ‘Rating’: {‘port’: ‘OutputPort0’, ‘type’: ‘Categorical’, ‘categories’: 2}}
{‘Image’: {‘port’: ‘InputPort0’, ‘type’: ‘Image’, ‘categories’: 277524}, ‘Rating’: {‘port’: ‘OutputPort0’, ‘type’: ‘Categorical’, ‘categories’: 2}}
{‘Image’: {‘port’: ‘InputPort0’, ‘type’: ‘Image’, ‘categories’: 277524}, ‘Rating’: {‘port’: ‘OutputPort0’, ‘type’: ‘Categorical’, ‘categories’: 2}}
{‘Image’: {‘port’: ‘InputPort0’, ‘type’: ‘Image’, ‘categories’: 277524}, ‘Rating’: {‘port’: ‘OutputPort0’, ‘type’: ‘Categorical’, ‘categories’: 2}}
{‘Image’: {‘port’: ‘InputPort0’, ‘type’: ‘Image’, ‘categories’: 277524}, ‘Rating’: {‘port’: ‘OutputPort0’, ‘type’: ‘Categorical’, ‘categories’: 2}}
{‘Image’: {‘port’: ‘InputPort0’, ‘type’: ‘Image’, ‘categories’: 277524}, ‘Rating’: {‘port’: ‘OutputPort0’, ‘type’: ‘Categorical’, ‘categories’: 2}}
{‘Image’: {‘port’: ‘InputPort0’, ‘type’: ‘Image’, ‘categories’: 277524}, ‘Rating’: {‘port’: ‘OutputPort0’, ‘type’: ‘Categorical’, ‘categories’: 2}}
{‘Image’: {‘port’: ‘InputPort0’, ‘type’: ‘Image’, ‘categories’: 277524}, ‘Rating’: {‘port’: ‘OutputPort0’, ‘type’: ‘Categorical’, ‘categories’: 2}}
{‘Image’: {‘port’: ‘InputPort0’, ‘type’: ‘Image’, ‘categories’: 277524}, ‘Rating’: {‘port’: ‘OutputPort0’, ‘type’: ‘Categorical’, ‘categories’: 2}}
{‘Image’: {‘port’: ‘InputPort0’, ‘type’: ‘Image’, ‘categories’: 277524}, ‘Rating’: {‘port’: ‘OutputPort0’, ‘type’: ‘Categorical’, ‘categories’: 2}}
{‘Image’: {‘port’: ‘InputPort0’, ‘type’: ‘Image’, ‘categories’: 277524}, ‘Rating’: {‘port’: ‘OutputPort0’, ‘type’: ‘Categorical’, ‘categories’: 2}}
{‘Image’: {‘port’: ‘InputPort0’, ‘type’: ‘Image’, ‘categories’: 277524}, ‘Rating’: {‘port’: ‘OutputPort0’, ‘type’: ‘Categorical’, ‘categories’: 2}}
{‘Image’: {‘port’: ‘InputPort0’, ‘type’: ‘Image’, ‘categories’: 277524}, ‘Rating’: {‘port’: ‘OutputPort0’, ‘type’: ‘Categorical’, ‘categories’: 2}}
{‘Image’: {‘port’: ‘InputPort0’, ‘type’: ‘Image’, ‘categories’: 277524}, ‘Rating’: {‘port’: ‘OutputPort0’, ‘type’: ‘Categorical’, ‘categories’: 2}}
{‘Image’: {‘port’: ‘InputPort0’, ‘type’: ‘Image’, ‘categories’: 277524}, ‘Rating’: {‘port’: ‘OutputPort0’, ‘type’: ‘Categorical’, ‘categories’: 2}}
{‘Image’: {‘port’: ‘InputPort0’, ‘type’: ‘Image’, ‘categories’: 277524}, ‘Rating’: {‘port’: ‘OutputPort0’, ‘type’: ‘Categorical’, ‘categories’: 2}}
{‘Image’: {‘port’: ‘InputPort0’, ‘type’: ‘Image’, ‘categories’: 277524}, ‘Rating’: {‘port’: ‘OutputPort0’, ‘type’: ‘Categorical’, ‘categories’: 2}}
{‘Image’: {‘port’: ‘InputPort0’, ‘type’: ‘Image’, ‘categories’: 277524}, ‘Rating’: {‘port’: ‘OutputPort0’, ‘type’: ‘Categorical’, ‘categories’: 2}}
{‘Image’: {‘port’: ‘InputPort0’, ‘type’: ‘Image’, ‘categories’: 277524}, ‘Rating’: {‘port’: ‘OutputPort0’, ‘type’: ‘Categorical’, ‘categories’: 2}}
{‘Image’: {‘port’: ‘InputPort0’, ‘type’: ‘Image’, ‘categories’: 277524}, ‘Rating’: {‘port’: ‘OutputPort0’, ‘type’: ‘Categorical’, ‘categories’: 2}}
{‘Image’: {‘port’: ‘InputPort0’, ‘type’: ‘Image’, ‘categories’: 277524}, ‘Rating’: {‘port’: ‘OutputPort0’, ‘type’: ‘Categorical’, ‘categories’: 2}}
{‘Image’: {‘port’: ‘InputPort0’, ‘type’: ‘Image’, ‘categories’: 277524}, ‘Rating’: {‘port’: ‘OutputPort0’, ‘type’: ‘Categorical’, ‘categories’: 2}}
{‘Image’: {‘port’: ‘InputPort0’, ‘type’: ‘Image’, ‘categories’: 277524}, ‘Rating’: {‘port’: ‘OutputPort0’, ‘type’: ‘Categorical’, ‘categories’: 2}}
{‘Image’: {‘port’: ‘InputPort0’, ‘type’: ‘Image’, ‘categories’: 277524}, ‘Rating’: {‘port’: ‘OutputPort0’, ‘type’: ‘Categorical’, ‘categories’: 2}}
{‘Image’: {‘port’: ‘InputPort0’, ‘type’: ‘Image’, ‘categories’: 277524}, ‘Rating’: {‘port’: ‘OutputPort0’, ‘type’: ‘Categorical’, ‘categories’: 2}}
{‘Image’: {‘port’: ‘InputPort0’, ‘type’: ‘Image’, ‘categories’: 277524}, ‘Rating’: {‘port’: ‘OutputPort0’, ‘type’: ‘Categorical’, ‘categories’: 2}}
{‘Image’: {‘port’: ‘InputPort0’, ‘type’: ‘Image’, ‘categories’: 277524}, ‘Rating’: {‘port’: ‘OutputPort0’, ‘type’: ‘Categorical’, ‘categories’: 2}}
{‘Image’: {‘port’: ‘InputPort0’, ‘type’: ‘Image’, ‘categories’: 277524}, ‘Rating’: {‘port’: ‘OutputPort0’, ‘type’: ‘Categorical’, ‘categories’: 2}}
{‘Image’: {‘port’: ‘InputPort0’, ‘type’: ‘Image’, ‘categories’: 277524}, ‘Rating’: {‘port’: ‘OutputPort0’, ‘type’: ‘Categorical’, ‘categories’: 2}}
{‘Image’: {‘port’: ‘InputPort0’, ‘type’: ‘Image’, ‘categories’: 277524}, ‘Rating’: {‘port’: ‘OutputPort0’, ‘type’: ‘Categorical’, ‘categories’: 2}}
{‘Image’: {‘port’: ‘InputPort0’, ‘type’: ‘Image’, ‘categories’: 277524}, ‘Rating’: {‘port’: ‘OutputPort0’, ‘type’: ‘Categorical’, ‘categories’: 2}}
{‘Image’: {‘port’: ‘InputPort0’, ‘type’: ‘Image’, ‘categories’: 277524}, ‘Rating’: {‘port’: ‘OutputPort0’, ‘type’: ‘Categorical’, ‘categories’: 2}}
Traceback (most recent call last):
File “project\network.py”, line 594, in project.network.getOutputShapes
File “project\network.py”, line 44, in project.network.jsonToGraph
KeyError: ‘target’
Traceback (most recent call last):
File “project\network.py”, line 594, in project.network.getOutputShapes
File “project\network.py”, line 44, in project.network.jsonToGraph
KeyError: ‘target’
Traceback (most recent call last):
File “project\network.py”, line 594, in project.network.getOutputShapes
File “project\network.py”, line 44, in project.network.jsonToGraph
KeyError: ‘target’
{‘Image’: {‘port’: ‘InputPort0’, ‘type’: ‘Image’, ‘categories’: 277524}, ‘Rating’: {‘port’: ‘OutputPort0’, ‘type’: ‘Categorical’, ‘categories’: 2}}
{‘Image’: {‘port’: ‘InputPort0’, ‘type’: ‘Image’, ‘categories’: 277524}, ‘Rating’: {‘port’: ‘OutputPort0’, ‘type’: ‘Categorical’, ‘categories’: 2}}
{‘Image’: {‘port’: ‘InputPort0’, ‘type’: ‘Image’, ‘categories’: 277524}, ‘Rating’: {‘port’: ‘OutputPort0’, ‘type’: ‘Categorical’, ‘categories’: 2}}
{‘Image’: {‘port’: ‘InputPort0’, ‘type’: ‘Image’, ‘categories’: 277524}, ‘Rating’: {‘port’: ‘OutputPort0’, ‘type’: ‘Categorical’, ‘categories’: 2}}
{‘Image’: {‘port’: ‘InputPort0’, ‘type’: ‘Image’, ‘categories’: 277524}, ‘Rating’: {‘port’: ‘OutputPort0’, ‘type’: ‘Categorical’, ‘categories’: 2}}
{‘Image’: {‘port’: ‘InputPort0’, ‘type’: ‘Image’, ‘categories’: 277524}, ‘Rating’: {‘port’: ‘OutputPort0’, ‘type’: ‘Categorical’, ‘categories’: 2}}
{‘Image’: {‘port’: ‘InputPort0’, ‘type’: ‘Image’, ‘categories’: 277524}, ‘Rating’: {‘port’: ‘OutputPort0’, ‘type’: ‘Categorical’, ‘categories’: 2}}
{‘Image’: {‘port’: ‘InputPort0’, ‘type’: ‘Image’, ‘categories’: 277524}, ‘Rating’: {‘port’: ‘OutputPort0’, ‘type’: ‘Categorical’, ‘categories’: 2}}
{‘Image’: {‘port’: ‘InputPort0’, ‘type’: ‘Image’, ‘categories’: 277524}, ‘Rating’: {‘port’: ‘OutputPort0’, ‘type’: ‘Categorical’, ‘categories’: 2}}
{‘Image’: {‘port’: ‘InputPort0’, ‘type’: ‘Image’, ‘categories’: 277524}, ‘Rating’: {‘port’: ‘OutputPort0’, ‘type’: ‘Categorical’, ‘categories’: 2}}
{‘Image’: {‘port’: ‘InputPort0’, ‘type’: ‘Image’, ‘categories’: 277524}, ‘Rating’: {‘port’: ‘OutputPort0’, ‘type’: ‘Categorical’, ‘categories’: 2}}
{‘Image’: {‘port’: ‘InputPort0’, ‘type’: ‘Image’, ‘categories’: 277524}, ‘Rating’: {‘port’: ‘OutputPort0’, ‘type’: ‘Categorical’, ‘categories’: 2}}
{‘Image’: {‘port’: ‘InputPort0’, ‘type’: ‘Image’, ‘categories’: 277524}, ‘Rating’: {‘port’: ‘OutputPort0’, ‘type’: ‘Categorical’, ‘categories’: 2}}
{‘Image’: {‘port’: ‘InputPort0’, ‘type’: ‘Image’, ‘categories’: 277524}, ‘Rating’: {‘port’: ‘OutputPort0’, ‘type’: ‘Categorical’, ‘categories’: 2}}
{‘Image’: {‘port’: ‘InputPort0’, ‘type’: ‘Image’, ‘categories’: 277524}, ‘Rating’: {‘port’: ‘OutputPort0’, ‘type’: ‘Categorical’, ‘categories’: 2}}
{‘Image’: {‘port’: ‘InputPort0’, ‘type’: ‘Image’, ‘categories’: 277524}, ‘Rating’: {‘port’: ‘OutputPort0’, ‘type’: ‘Categorical’, ‘categories’: 2}}
{‘Image’: {‘port’: ‘InputPort0’, ‘type’: ‘Image’, ‘categories’: 277524}, ‘Rating’: {‘port’: ‘OutputPort0’, ‘type’: ‘Categorical’, ‘categories’: 2}}
{‘Image’: {‘port’: ‘InputPort0’, ‘type’: ‘Image’, ‘categories’: 277524}, ‘Rating’: {‘port’: ‘OutputPort0’, ‘type’: ‘Categorical’, ‘categories’: 2}}
{‘Image’: {‘port’: ‘InputPort0’, ‘type’: ‘Image’, ‘categories’: 277524}, ‘Rating’: {‘port’: ‘OutputPort0’, ‘type’: ‘Categorical’, ‘categories’: 2}}
{‘Image’: {‘port’: ‘InputPort0’, ‘type’: ‘Image’, ‘categories’: 277524}, ‘Rating’: {‘port’: ‘OutputPort0’, ‘type’: ‘Categorical’, ‘categories’: 2}}
{‘Image’: {‘port’: ‘InputPort0’, ‘type’: ‘Image’, ‘categories’: 277524}, ‘Rating’: {‘port’: ‘OutputPort0’, ‘type’: ‘Categorical’, ‘categories’: 2}}
{‘Image’: {‘port’: ‘InputPort0’, ‘type’: ‘Image’, ‘categories’: 277524}, ‘Rating’: {‘port’: ‘OutputPort0’, ‘type’: ‘Categorical’, ‘categories’: 2}}
{‘Image’: {‘port’: ‘InputPort0’, ‘type’: ‘Image’, ‘categories’: 277524}, ‘Rating’: {‘port’: ‘OutputPort0’, ‘type’: ‘Categorical’, ‘categories’: 2}}
{‘Image’: {‘port’: ‘InputPort0’, ‘type’: ‘Image’, ‘categories’: 277524}, ‘Rating’: {‘port’: ‘OutputPort0’, ‘type’: ‘Categorical’, ‘categories’: 2}}
{‘Image’: {‘port’: ‘InputPort0’, ‘type’: ‘Image’, ‘categories’: 277524}, ‘Rating’: {‘port’: ‘OutputPort0’, ‘type’: ‘Categorical’, ‘categories’: 2}}
{‘Image’: {‘port’: ‘InputPort0’, ‘type’: ‘Image’, ‘categories’: 277524}, ‘Rating’: {‘port’: ‘OutputPort0’, ‘type’: ‘Categorical’, ‘categories’: 2}}
{‘Image’: {‘port’: ‘InputPort0’, ‘type’: ‘Image’, ‘categories’: 277524}, ‘Rating’: {‘port’: ‘OutputPort0’, ‘type’: ‘Categorical’, ‘categories’: 2}}
{‘Image’: {‘port’: ‘InputPort0’, ‘type’: ‘Image’, ‘categories’: 277524}, ‘Rating’: {‘port’: ‘OutputPort0’, ‘type’: ‘Categorical’, ‘categories’: 2}}
{‘Image’: {‘port’: ‘InputPort0’, ‘type’: ‘Image’, ‘categories’: 277524}, ‘Rating’: {‘port’: ‘OutputPort0’, ‘type’: ‘Categorical’, ‘categories’: 2}}
{‘Image’: {‘port’: ‘InputPort0’, ‘type’: ‘Image’, ‘categories’: 277524}, ‘Rating’: {‘port’: ‘OutputPort0’, ‘type’: ‘Categorical’, ‘categories’: 2}}
WARNING:django.request:Not Found: /favicon.ico

starting training process
{‘command’: ‘start’, ‘compute-server’: ‘127.0.0.1:22001’, ‘webapp-server’: ‘127.0.0.1:22002’, ‘load_weights’: ‘-1’, ‘save_weights_on’: ‘epoch’, ‘devices’: ‘[{“compute_capabilty”:5,“id”:0,“index”:1,“mem_free”:2107703296,“mem_total”:2147483648,“name”:“GeForce GTX 960M”,“percent”:1.8524169921875,“type”:“GPU”}]’, ‘run_name’: ‘Run15’, ‘project_id’: ‘0’, ‘user_email’: ‘james.jayeola@stu.cu.edu.ng’, ‘username’: ‘Jamiepoo17’, ‘run_session_id’: ‘qkwbIZxnYojv96bsqWgWxCD3LQR0MQYa’, ‘user_id’: ‘1’, ‘run_id’: ‘15’, ‘project_type’: ‘CUSTOM_NN’}
[{‘compute_capabilty’: 5, ‘id’: 0, ‘index’: 1, ‘mem_free’: 2107703296, ‘mem_total’: 2147483648, ‘name’: ‘GeForce GTX 960M’, ‘percent’: 1.8524169921875, ‘type’: ‘GPU’}]
Runing on decives : [’/device:GPU:0’]
2020-08-03 04:31:48.614118: W tensorflow/core/common_runtime/gpu/gpu_bfc_allocator.cc:39] Overriding allow_growth setting because the TF_FORCE_GPU_ALLOW_GROWTH environment variable is set. Original config value was 0.
2020-08-03 04:31:52.066076: W tensorflow/stream_executor/gpu/redzone_allocator.cc:312] Internal: Invoking GPU asm compilation is supported on Cuda non-Windows platforms only
Relying on driver to perform ptx compilation. This message will be only logged once.
2020-08-03 04:31:52.110267: W tensorflow/core/common_runtime/bfc_allocator.cc:243] Allocator (GPU_0_bfc) ran out of memory trying to allocate 1.03GiB with freed_by_count=0. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available.
2020-08-03 04:31:52.190501: W tensorflow/core/common_runtime/bfc_allocator.cc:243] Allocator (GPU_0_bfc) ran out of memory trying to allocate 752.11MiB with freed_by_count=0. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available.
2020-08-03 04:31:52.192960: W tensorflow/core/common_runtime/bfc_allocator.cc:243] Allocator (GPU_0_bfc) ran out of memory trying to allocate 328.50MiB with freed_by_count=0. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available.
2020-08-03 04:31:52.193538: W tensorflow/core/kernels/gpu_utils.cc:48] Failed to allocate memory for convolution redzone checking; skipping this check. This is benign and only means that we won’t check cudnn for out-of-bounds reads and writes. This message will only be printed once.
2020-08-03 04:31:52.533398: W tensorflow/core/common_runtime/bfc_allocator.cc:243] Allocator (GPU_0_bfc) ran out of memory trying to allocate 2.76GiB with freed_by_count=0. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available.
2020-08-03 04:31:52.533934: W tensorflow/core/common_runtime/bfc_allocator.cc:243] Allocator (GPU_0_bfc) ran out of memory trying to allocate 1.54GiB with freed_by_count=0. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available.
2020-08-03 04:31:52.691045: W tensorflow/core/common_runtime/bfc_allocator.cc:243] Allocator (GPU_0_bfc) ran out of memory trying to allocate 1.39GiB with freed_by_count=0. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available.
2020-08-03 04:31:52.890169: W tensorflow/core/common_runtime/bfc_allocator.cc:243] Allocator (GPU_0_bfc) ran out of memory trying to allocate 719.13MiB with freed_by_count=0. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available.
2020-08-03 04:31:52.890704: W tensorflow/core/common_runtime/bfc_allocator.cc:309] Garbage collection: deallocate free memory regions (i.e., allocations) so that we can re-allocate a larger region to avoid OOM due to memory fragmentation. If you see this message frequently, you are running near the threshold of the available device memory and re-allocation may incur great performance overhead. You may try smaller batch sizes to observe the performance impact. Set TF_ENABLE_GPU_GARBAGE_COLLECTION=false if you’d like to disable this feature.
2020-08-03 04:31:53.358481: W tensorflow/core/common_runtime/bfc_allocator.cc:243] Allocator (GPU_0_bfc) ran out of memory trying to allocate 1.39GiB with freed_by_count=0. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available.
2020-08-03 04:31:53.359179: W tensorflow/core/common_runtime/bfc_allocator.cc:243] Allocator (GPU_0_bfc) ran out of memory trying to allocate 1.05GiB with freed_by_count=0. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available.
2020-08-03 04:31:53.499187: W tensorflow/core/common_runtime/bfc_allocator.cc:243] Allocator (GPU_0_bfc) ran out of memory trying to allocate 778.75MiB with freed_by_count=0. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available.
2020-08-03 04:31:56.043560: W tensorflow/core/common_runtime/bfc_allocator.cc:309] Garbage collection: deallocate free memory regions (i.e., allocations) so that we can re-allocate a larger region to avoid OOM due to memory fragmentation. If you see this message frequently, you are running near the threshold of the available device memory and re-allocation may incur great performance overhead. You may try smaller batch sizes to observe the performance impact. Set TF_ENABLE_GPU_GARBAGE_COLLECTION=false if you’d like to disable this feature.
2020-08-03 04:59:08.213233: W tensorflow/core/kernels/data/generator_dataset_op.cc:103] Error occurred when finalizing GeneratorDataset iterator: Cancelled: Operation was cancelled
2020-08-03 05:14:20.538415: W tensorflow/core/kernels/data/generator_dataset_op.cc:103] Error occurred when finalizing GeneratorDataset iterator: Cancelled: Operation was cancelled
2020-08-03 05:30:09.527570: W tensorflow/core/kernels/data/generator_dataset_op.cc:103] Error occurred when finalizing GeneratorDataset iterator: Cancelled: Operation was cancelled
2020-08-03 05:45:17.118490: W tensorflow/core/kernels/data/generator_dataset_op.cc:103] Error occurred when finalizing GeneratorDataset iterator: Cancelled: Operation was cancelled
2020-08-03 06:00:22.232961: W tensorflow/core/kernels/data/generator_dataset_op.cc:103] Error occurred when finalizing GeneratorDataset iterator: Cancelled: Operation was cancelled
2020-08-03 06:15:19.432184: W tensorflow/core/kernels/data/generator_dataset_op.cc:103] Error occurred when finalizing GeneratorDataset iterator: Cancelled: Operation was cancelled
2020-08-03 06:30:09.430405: W tensorflow/core/kernels/data/generator_dataset_op.cc:103] Error occurred when finalizing GeneratorDataset iterator: Cancelled: Operation was cancelled
2020-08-03 06:44:54.037853: W tensorflow/core/kernels/data/generator_dataset_op.cc:103] Error occurred when finalizing GeneratorDataset iterator: Cancelled: Operation was cancelled
2020-08-03 06:59:31.413415: W tensorflow/core/kernels/data/generator_dataset_op.cc:103] Error occurred when finalizing GeneratorDataset iterator: Cancelled: Operation was cancelled
2020-08-03 07:14:07.909873: W tensorflow/core/kernels/data/generator_dataset_op.cc:103] Error occurred when finalizing GeneratorDataset iterator: Cancelled: Operation was cancelled
2020-08-03 07:28:43.507343: W tensorflow/core/kernels/data/generator_dataset_op.cc:103] Error occurred when finalizing GeneratorDataset iterator: Cancelled: Operation was cancelled
2020-08-03 07:43:21.220806: W tensorflow/core/kernels/data/generator_dataset_op.cc:103] Error occurred when finalizing GeneratorDataset iterator: Cancelled: Operation was cancelled
2020-08-03 07:57:54.645278: W tensorflow/core/kernels/data/generator_dataset_op.cc:103] Error occurred when finalizing GeneratorDataset iterator: Cancelled: Operation was cancelled
2020-08-03 08:12:30.264840: W tensorflow/core/kernels/data/generator_dataset_op.cc:103] Error occurred when finalizing GeneratorDataset iterator: Cancelled: Operation was cancelled
2020-08-03 08:15:03.794198: W tensorflow/core/kernels/data/generator_dataset_op.cc:103] Error occurred when finalizing GeneratorDataset iterator: Cancelled: Operation was cancelled
2020-08-03 08:15:03.939896: W tensorflow/core/kernels/data/generator_dataset_op.cc:103] Error occurred when finalizing GeneratorDataset iterator: Cancelled: Operation was cancelled
2020-08-03 10:57:16.713497: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library cudart64_101.dll

Your “HMHNGNG” model (version: 1) has been successfully deployed.
You can now access it at the following endpoint:

	/models/HMHNGNG/v1/predict

2020-08-03 04:02:28,533 INFO SUCCESS: The process with PID 7696 (child process of PID 8128) has been terminated.
2020-08-03 04:02:28,534 INFO SUCCESS: The process with PID 8128 (child process of PID 15756) has been terminated.
2020-08-03 04:02:28,534 INFO SUCCESS: The process with PID 15756 (child process of PID 10964) has been terminated.
2020-08-03 04:02:28,535 INFO stopped: DLS_CVAT_server (exit status 1)
2020-08-03 04:02:28,635 INFO SUCCESS: The process with PID 6484 (child process of PID 9956) has been terminated.
2020-08-03 04:02:28,635 INFO SUCCESS: The process with PID 9956 (child process of PID 16044) has been terminated.
2020-08-03 04:02:28,635 INFO SUCCESS: The process with PID 16044 (child process of PID 4604) has been terminated.
2020-08-03 04:02:28,635 INFO SUCCESS: The process with PID 4604 (child process of PID 1248) has been terminated.
2020-08-03 04:02:28,635 INFO SUCCESS: The process with PID 1248 (child process of PID 14460) has been terminated.
2020-08-03 04:02:28,635 INFO SUCCESS: The process with PID 14460 (child process of PID 12496) has been terminated.
2020-08-03 04:02:28,635 INFO SUCCESS: The process with PID 12496 (child process of PID 10964) has been terminated.
2020-08-03 04:02:28,635 INFO stopped: DLS_Compute_Server (exit status 1)
2020-08-03 04:02:28,720 INFO SUCCESS: The process with PID 14184 (child process of PID 13720) has been terminated.
2020-08-03 04:02:28,720 INFO SUCCESS: The process with PID 13720 (child process of PID 14964) has been terminated.
2020-08-03 04:02:28,720 INFO SUCCESS: The process with PID 14964 (child process of PID 13908) has been terminated.
2020-08-03 04:02:28,720 INFO SUCCESS: The process with PID 13908 (child process of PID 3328) has been terminated.
2020-08-03 04:02:28,720 INFO SUCCESS: The process with PID 3328 (child process of PID 10964) has been terminated.
2020-08-03 04:02:28,720 INFO stopped: DLS_Monitor_Server (exit status 1)
2020-08-03 04:02:28,812 INFO SUCCESS: The process with PID 11068 (child process of PID 9356) has been terminated.
2020-08-03 04:02:28,812 INFO SUCCESS: The process with PID 9356 (child process of PID 7592) has been terminated.
2020-08-03 04:02:28,812 INFO SUCCESS: The process with PID 7592 (child process of PID 4052) has been terminated.
2020-08-03 04:02:28,813 INFO SUCCESS: The process with PID 4052 (child process of PID 3412) has been terminated.
2020-08-03 04:02:28,813 INFO SUCCESS: The process with PID 3412 (child process of PID 10964) has been terminated.
2020-08-03 04:02:28,813 INFO stopped: DLS_WebApp_Server (exit status 1)
2020-08-03 04:02:28,900 INFO SUCCESS: The process with PID 7720 (child process of PID 9240) has been terminated.
2020-08-03 04:02:28,901 INFO SUCCESS: The process with PID 9240 (child process of PID 11960) has been terminated.
2020-08-03 04:02:28,901 INFO SUCCESS: The process with PID 11960 (child process of PID 2636) has been terminated.
2020-08-03 04:02:28,901 INFO SUCCESS: The process with PID 2636 (child process of PID 4392) has been terminated.
2020-08-03 04:02:28,901 INFO SUCCESS: The process with PID 4392 (child process of PID 2064) has been terminated.
2020-08-03 04:02:28,901 INFO SUCCESS: The process with PID 2064 (child process of PID 10964) has been terminated.
2020-08-03 04:02:28,901 INFO stopped: Jupyter-Lab (exit status 1)
2020-08-03 04:02:28,985 INFO SUCCESS: The process with PID 4576 (child process of PID 3196) has been terminated.
2020-08-03 04:02:28,985 INFO SUCCESS: The process with PID 3196 (child process of PID 392) has been terminated.
2020-08-03 04:02:28,985 INFO SUCCESS: The process with PID 392 (child process of PID 2412) has been terminated.
2020-08-03 04:02:28,985 INFO SUCCESS: The process with PID 2412 (child process of PID 15136) has been terminated.
2020-08-03 04:02:28,985 INFO SUCCESS: The process with PID 15136 (child process of PID 10964) has been terminated.
2020-08-03 04:02:28,986 INFO stopped: Redis (exit status 1)
2020-08-03 04:02:29,101 INFO SUCCESS: The process with PID 3164 (child process of PID 3684) has been terminated.
2020-08-03 04:02:29,101 INFO SUCCESS: The process with PID 3684 (child process of PID 3532) has been terminated.
2020-08-03 04:02:29,101 INFO SUCCESS: The process with PID 3532 (child process of PID 7024) has been terminated.
2020-08-03 04:02:29,101 INFO SUCCESS: The process with PID 7024 (child process of PID 13180) has been terminated.
2020-08-03 04:02:29,101 INFO SUCCESS: The process with PID 13180 (child process of PID 6272) has been terminated.
2020-08-03 04:02:29,102 INFO SUCCESS: The process with PID 6272 (child process of PID 3672) has been terminated.
2020-08-03 04:02:29,102 INFO SUCCESS: The process with PID 3672 (child process of PID 10964) has been terminated.
2020-08-03 04:02:29,102 INFO stopped: nginx (exit status 1)
2020-08-03 04:03:02,891 INFO process group added: ‘DLS_CVAT_server’
2020-08-03 04:03:02,892 INFO process group added: ‘DLS_Compute_Server’
2020-08-03 04:03:02,892 INFO process group added: ‘DLS_Deployment_Server’
2020-08-03 04:03:02,893 INFO process group added: ‘DLS_Monitor_Server’
2020-08-03 04:03:02,893 INFO process group added: ‘DLS_WebApp_Server’
2020-08-03 04:03:02,894 INFO process group added: ‘Jupyter-Lab’
2020-08-03 04:03:02,894 INFO process group added: ‘Redis’
2020-08-03 04:03:02,895 INFO process group added: ‘nginx’
2020-08-03 04:03:02,909 INFO RPC interface ‘supervisor’ initialized
2020-08-03 04:03:02,910 CRIT Server ‘inet_http_server’ running without any HTTP authentication checking
2020-08-03 04:03:02,910 INFO supervisord started with pid 2368
2020-08-03 04:03:14,942 INFO Spawned: ‘DLS_CVAT_server’ with pid 13512
2020-08-03 04:03:14,947 INFO Spawned: ‘DLS_Compute_Server’ with pid 8540
2020-08-03 04:03:14,953 INFO Spawned: ‘DLS_Deployment_Server’ with pid 12176
2020-08-03 04:03:14,957 INFO Spawned: ‘DLS_Monitor_Server’ with pid 6524
2020-08-03 04:03:14,962 INFO Spawned: ‘DLS_WebApp_Server’ with pid 14704
2020-08-03 04:03:14,968 INFO Spawned: ‘Jupyter-Lab’ with pid 9048
2020-08-03 04:03:14,974 INFO Spawned: ‘Redis’ with pid 16168
2020-08-03 04:03:14,980 INFO Spawned: ‘nginx’ with pid 12892
2020-08-03 04:03:16,019 INFO success: DLS_CVAT_server entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2020-08-03 04:03:16,020 INFO success: DLS_Compute_Server entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2020-08-03 04:03:16,020 INFO success: DLS_Deployment_Server entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2020-08-03 04:03:16,020 INFO success: DLS_Monitor_Server entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2020-08-03 04:03:16,020 INFO success: DLS_WebApp_Server entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2020-08-03 04:03:16,020 INFO success: Jupyter-Lab entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2020-08-03 04:03:16,020 INFO success: Redis entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2020-08-03 04:03:16,020 INFO success: nginx entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2020-08-03 04:03:22,628 INFO exited: DLS_Deployment_Server (termination normal; expected)

Hi @Jamiepoo17,

I have checked your logs and it looks like deployment server and CVAT is not installed properly. Please try to re-install the DLS.

Thanks,
Sandhaya