Newer
Older
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
600
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
700
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
756
757
758
759
760
761
762
763
764
765
766
767
768
769
770
771
772
773
774
775
776
777
778
779
780
781
782
783
784
785
786
787
788
789
790
791
792
793
794
795
796
797
798
799
800
801
802
803
804
805
806
807
808
809
810
811
812
813
814
815
816
817
818
819
820
821
822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
841
842
843
844
845
846
847
848
849
850
851
852
853
854
855
856
857
858
859
860
861
862
863
864
865
866
867
868
869
870
871
872
873
874
875
876
877
878
879
880
881
882
883
884
885
886
887
888
889
890
891
892
893
894
895
896
897
898
899
900
901
902
903
904
905
906
907
908
909
910
911
912
913
914
915
916
917
918
919
920
921
922
923
924
925
926
927
928
929
930
931
932
933
934
935
936
937
938
939
940
941
942
943
944
945
946
947
948
949
950
951
952
953
954
955
956
957
958
959
960
961
962
963
964
965
966
967
968
969
970
971
972
973
974
975
976
977
978
979
980
981
982
983
984
985
986
987
988
989
990
991
992
993
994
995
996
997
998
999
1000
2023-11-23 19:27:09.268020: I tensorflow/core/platform/cpu_feature_guard.cc:182] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.
To enable the following instructions: AVX2 AVX512F FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags.
Using TensorFlow backend
Namespace(data_input='imu+bvp', feature_method='None', lbl_str='pss', model='cnn1d', overwrite=0, subject=3, test_standing=1, train_len=5, win_shift=0.2, win_size=12)
unable to find matching config id
Data id not set, auto assigned to: 1
imu-bvp_rr_Pilot03_id1_combi5.0-7.0-10.0-12.0-15.0
train
(36036, 11)
test
(287283, 10)
---CNN1D---
2023-11-23 19:27:47.489396: I tensorflow/compiler/xla/stream_executor/cuda/cuda_gpu_executor.cc:995] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355
2023-11-23 19:27:47.830784: I tensorflow/compiler/xla/stream_executor/cuda/cuda_gpu_executor.cc:995] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355
2023-11-23 19:27:47.831037: I tensorflow/compiler/xla/stream_executor/cuda/cuda_gpu_executor.cc:995] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355
input shape: (1440, 7)
x shape: (121, 1440, 7)
2023-11-23 19:27:47.850993: I tensorflow/compiler/xla/stream_executor/cuda/cuda_gpu_executor.cc:995] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355
2023-11-23 19:27:47.851215: I tensorflow/compiler/xla/stream_executor/cuda/cuda_gpu_executor.cc:995] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355
2023-11-23 19:27:47.851368: I tensorflow/compiler/xla/stream_executor/cuda/cuda_gpu_executor.cc:995] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355
2023-11-23 19:27:48.369936: I tensorflow/compiler/xla/stream_executor/cuda/cuda_gpu_executor.cc:995] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355
2023-11-23 19:27:48.370212: I tensorflow/compiler/xla/stream_executor/cuda/cuda_gpu_executor.cc:995] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355
2023-11-23 19:27:48.370369: I tensorflow/compiler/xla/stream_executor/cuda/cuda_gpu_executor.cc:995] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero. See more at https://github.com/torvalds/linux/blob/v6.0/Documentation/ABI/testing/sysfs-bus-pci#L344-L355
2023-11-23 19:27:48.370506: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1639] Created device /job:localhost/replica:0/task:0/device:GPU:0 with 14602 MB memory: -> device: 0, name: Quadro RTX 5000, pci bus id: 0000:0b:00.0, compute capability: 7.5
Search: Running Trial #1
Value |Best Value So Far |Hyperparameter
5 |5 |n_layers
64 |64 |filter_unit0
32 |32 |filter_unit1
128 |128 |filter_unit2
128 |128 |filter_unit3
128 |128 |filter_unit4
2 |2 |kernel_size0
4 |4 |kernel_size1
5 |5 |kernel_size2
2 |2 |kernel_size3
2 |2 |kernel_size4
2 |2 |pool_size0
2 |2 |pool_size1
1 |1 |pool_size2
4 |4 |pool_size3
4 |4 |pool_size4
1 |1 |stride_size0
3 |3 |stride_size1
4 |4 |stride_size2
4 |4 |stride_size3
4 |4 |stride_size4
0.3 |0.3 |dropout0
0.2 |0.2 |dropout1
0.3 |0.3 |dropout2
0.2 |0.2 |dropout3
0.1 |0.1 |dropout4
2023-11-23 19:27:57.559717: I tensorflow/compiler/xla/stream_executor/cuda/cuda_dnn.cc:432] Loaded cuDNN version 8600
2023-11-23 19:28:03.154767: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x1495926eeda0 initialized for platform CUDA (this does not guarantee that XLA will be used). Devices:
2023-11-23 19:28:03.154806: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): Quadro RTX 5000, Compute Capability 7.5
2023-11-23 19:28:04.140909: I tensorflow/compiler/mlir/tensorflow/utils/dump_mlir_util.cc:255] disabling MLIR crash reproducer, set env var `MLIR_CRASH_REPRODUCER_DIRECTORY` to enable.
2023-11-23 19:28:09.171523: I ./tensorflow/compiler/jit/device_compiler.h:186] Compiled cluster using XLA! This line is logged at most once for the lifetime of the process.
[2K
[2K
Trial 1 Complete [00h 00m 26s]
val_loss: 14.554520606994629
Best val_loss So Far: 14.554520606994629
Total elapsed time: 00h 00m 26s
Search: Running Trial #2
Value |Best Value So Far |Hyperparameter
2 |5 |n_layers
64 |64 |filter_unit0
128 |32 |filter_unit1
3 |2 |kernel_size0
1 |4 |kernel_size1
4 |2 |pool_size0
4 |2 |pool_size1
1 |1 |stride_size0
3 |3 |stride_size1
0 |0.3 |dropout0
0.3 |0.2 |dropout1
WARNING:tensorflow:Callback method `on_train_batch_end` is slow compared to the batch time (batch time: 0.0031s vs `on_train_batch_end` time: 0.0043s). Check your callbacks.
Callback method `on_train_batch_end` is slow compared to the batch time (batch time: 0.0031s vs `on_train_batch_end` time: 0.0043s). Check your callbacks.
[2K
[2K
Trial 2 Complete [00h 00m 03s]
val_loss: 14.867725372314453
Best val_loss So Far: 14.554520606994629
Total elapsed time: 00h 00m 30s
Search: Running Trial #3
Value |Best Value So Far |Hyperparameter
5 |5 |n_layers
128 |64 |filter_unit0
128 |32 |filter_unit1
256 |128 |filter_unit2
128 |128 |filter_unit3
128 |128 |filter_unit4
5 |2 |kernel_size0
5 |4 |kernel_size1
4 |5 |kernel_size2
2 |2 |kernel_size3
4 |2 |kernel_size4
2 |2 |pool_size0
1 |2 |pool_size1
2 |1 |pool_size2
4 |4 |pool_size3
3 |4 |pool_size4
2 |1 |stride_size0
3 |3 |stride_size1
3 |4 |stride_size2
2 |4 |stride_size3
2 |4 |stride_size4
0.1 |0.3 |dropout0
0.3 |0.2 |dropout1
0.4 |0.3 |dropout2
0.4 |0.2 |dropout3
0.2 |0.1 |dropout4
2023-11-23 19:28:21.436743: I tensorflow/compiler/xla/stream_executor/cuda/cuda_driver.cc:753] failed to allocate 8.00GiB (8589934592 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2023-11-23 19:28:21.437614: I tensorflow/compiler/xla/stream_executor/cuda/cuda_driver.cc:753] failed to allocate 7.20GiB (7730940928 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2023-11-23 19:28:21.438447: I tensorflow/compiler/xla/stream_executor/cuda/cuda_driver.cc:753] failed to allocate 6.48GiB (6957846528 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2023-11-23 19:28:21.439272: I tensorflow/compiler/xla/stream_executor/cuda/cuda_driver.cc:753] failed to allocate 5.83GiB (6262061568 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2023-11-23 19:28:21.440098: I tensorflow/compiler/xla/stream_executor/cuda/cuda_driver.cc:753] failed to allocate 5.25GiB (5635855360 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2023-11-23 19:28:21.440914: I tensorflow/compiler/xla/stream_executor/cuda/cuda_driver.cc:753] failed to allocate 4.72GiB (5072269824 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2023-11-23 19:28:21.441736: I tensorflow/compiler/xla/stream_executor/cuda/cuda_driver.cc:753] failed to allocate 4.25GiB (4565042688 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2023-11-23 19:28:21.441750: W tensorflow/tsl/framework/bfc_allocator.cc:366] Garbage collection: deallocate free memory regions (i.e., allocations) so that we can re-allocate a larger region to avoid OOM due to memory fragmentation. If you see this message frequently, you are running near the threshold of the available device memory and re-allocation may incur great performance overhead. You may try smaller batch sizes to observe the performance impact. Set TF_ENABLE_GPU_GARBAGE_COLLECTION=false if you'd like to disable this feature.
2023-11-23 19:28:21.549183: I tensorflow/compiler/xla/stream_executor/cuda/cuda_driver.cc:753] failed to allocate 8.00GiB (8589934592 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2023-11-23 19:28:21.549215: W tensorflow/tsl/framework/bfc_allocator.cc:296] Allocator (GPU_0_bfc) ran out of memory trying to allocate 4.08GiB with freed_by_count=0. The caller indicates that this is not a failure, but this may mean that there could be performance gains if more memory were available.
2023-11-23 19:28:21.550336: I tensorflow/compiler/xla/stream_executor/cuda/cuda_driver.cc:753] failed to allocate 8.00GiB (8589934592 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2023-11-23 19:28:21.550350: W tensorflow/tsl/framework/bfc_allocator.cc:296] Allocator (GPU_0_bfc) ran out of memory trying to allocate 4.08GiB with freed_by_count=0. The caller indicates that this is not a failure, but this may mean that there could be performance gains if more memory were available.
2023-11-23 19:28:21.558730: I tensorflow/compiler/xla/stream_executor/cuda/cuda_driver.cc:753] failed to allocate 8.00GiB (8589934592 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2023-11-23 19:28:21.558757: W tensorflow/tsl/framework/bfc_allocator.cc:296] Allocator (GPU_0_bfc) ran out of memory trying to allocate 4.59GiB with freed_by_count=0. The caller indicates that this is not a failure, but this may mean that there could be performance gains if more memory were available.
2023-11-23 19:28:21.559843: I tensorflow/compiler/xla/stream_executor/cuda/cuda_driver.cc:753] failed to allocate 8.00GiB (8589934592 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2023-11-23 19:28:21.559861: W tensorflow/tsl/framework/bfc_allocator.cc:296] Allocator (GPU_0_bfc) ran out of memory trying to allocate 4.59GiB with freed_by_count=0. The caller indicates that this is not a failure, but this may mean that there could be performance gains if more memory were available.
2023-11-23 19:28:21.943362: I tensorflow/compiler/xla/stream_executor/cuda/cuda_driver.cc:753] failed to allocate 8.00GiB (8589934592 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2023-11-23 19:28:21.943406: W tensorflow/tsl/framework/bfc_allocator.cc:296] Allocator (GPU_0_bfc) ran out of memory trying to allocate 60.34MiB with freed_by_count=0. The caller indicates that this is not a failure, but this may mean that there could be performance gains if more memory were available.
2023-11-23 19:28:21.981522: I tensorflow/compiler/xla/stream_executor/cuda/cuda_driver.cc:753] failed to allocate 8.00GiB (8589934592 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2023-11-23 19:28:21.981555: W tensorflow/tsl/framework/bfc_allocator.cc:296] Allocator (GPU_0_bfc) ran out of memory trying to allocate 60.34MiB with freed_by_count=0. The caller indicates that this is not a failure, but this may mean that there could be performance gains if more memory were available.
WARNING:tensorflow:Callback method `on_train_batch_end` is slow compared to the batch time (batch time: 0.0054s vs `on_train_batch_end` time: 0.0062s). Check your callbacks.
Callback method `on_train_batch_end` is slow compared to the batch time (batch time: 0.0054s vs `on_train_batch_end` time: 0.0062s). Check your callbacks.
[2K
[2K
Trial 3 Complete [00h 00m 07s]
val_loss: 13.035322189331055
Best val_loss So Far: 13.035322189331055
Total elapsed time: 00h 00m 36s
Search: Running Trial #4
Value |Best Value So Far |Hyperparameter
2 |5 |n_layers
32 |128 |filter_unit0
32 |128 |filter_unit1
3 |5 |kernel_size0
3 |5 |kernel_size1
4 |2 |pool_size0
5 |1 |pool_size1
4 |2 |stride_size0
4 |3 |stride_size1
0.2 |0.1 |dropout0
0.4 |0.3 |dropout1
[2K
[2K
Trial 4 Complete [00h 00m 03s]
val_loss: 14.519936561584473
Best val_loss So Far: 13.035322189331055
Total elapsed time: 00h 00m 39s
Search: Running Trial #5
Value |Best Value So Far |Hyperparameter
3 |5 |n_layers
128 |128 |filter_unit0
32 |128 |filter_unit1
256 |256 |filter_unit2
1 |5 |kernel_size0
5 |5 |kernel_size1
1 |4 |kernel_size2
4 |2 |pool_size0
1 |1 |pool_size1
3 |2 |pool_size2
2 |2 |stride_size0
3 |3 |stride_size1
5 |3 |stride_size2
0 |0.1 |dropout0
0 |0.3 |dropout1
0.4 |0.4 |dropout2
2023-11-23 19:28:29.474895: I tensorflow/compiler/xla/stream_executor/cuda/cuda_driver.cc:753] failed to allocate 8.00GiB (8589934592 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2023-11-23 19:28:29.474934: W tensorflow/tsl/framework/bfc_allocator.cc:366] Garbage collection: deallocate free memory regions (i.e., allocations) so that we can re-allocate a larger region to avoid OOM due to memory fragmentation. If you see this message frequently, you are running near the threshold of the available device memory and re-allocation may incur great performance overhead. You may try smaller batch sizes to observe the performance impact. Set TF_ENABLE_GPU_GARBAGE_COLLECTION=false if you'd like to disable this feature.
2023-11-23 19:28:29.481137: I tensorflow/compiler/xla/stream_executor/cuda/cuda_driver.cc:753] failed to allocate 8.00GiB (8589934592 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2023-11-23 19:28:29.481167: W tensorflow/tsl/framework/bfc_allocator.cc:296] Allocator (GPU_0_bfc) ran out of memory trying to allocate 144.00MiB with freed_by_count=0. The caller indicates that this is not a failure, but this may mean that there could be performance gains if more memory were available.
2023-11-23 19:28:29.482277: I tensorflow/compiler/xla/stream_executor/cuda/cuda_driver.cc:753] failed to allocate 8.00GiB (8589934592 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2023-11-23 19:28:29.482291: W tensorflow/tsl/framework/bfc_allocator.cc:296] Allocator (GPU_0_bfc) ran out of memory trying to allocate 16.01MiB with freed_by_count=0. The caller indicates that this is not a failure, but this may mean that there could be performance gains if more memory were available.
2023-11-23 19:28:29.483374: I tensorflow/compiler/xla/stream_executor/cuda/cuda_driver.cc:753] failed to allocate 8.00GiB (8589934592 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2023-11-23 19:28:29.483386: W tensorflow/tsl/framework/bfc_allocator.cc:296] Allocator (GPU_0_bfc) ran out of memory trying to allocate 16.01MiB with freed_by_count=0. The caller indicates that this is not a failure, but this may mean that there could be performance gains if more memory were available.
2023-11-23 19:28:29.484464: I tensorflow/compiler/xla/stream_executor/cuda/cuda_driver.cc:753] failed to allocate 8.00GiB (8589934592 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2023-11-23 19:28:29.484476: W tensorflow/tsl/framework/bfc_allocator.cc:296] Allocator (GPU_0_bfc) ran out of memory trying to allocate 16.01MiB with freed_by_count=0. The caller indicates that this is not a failure, but this may mean that there could be performance gains if more memory were available.
2023-11-23 19:28:29.485553: I tensorflow/compiler/xla/stream_executor/cuda/cuda_driver.cc:753] failed to allocate 8.00GiB (8589934592 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2023-11-23 19:28:29.486643: I tensorflow/compiler/xla/stream_executor/cuda/cuda_driver.cc:753] failed to allocate 8.00GiB (8589934592 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2023-11-23 19:28:29.487723: I tensorflow/compiler/xla/stream_executor/cuda/cuda_driver.cc:753] failed to allocate 8.00GiB (8589934592 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2023-11-23 19:28:29.488805: I tensorflow/compiler/xla/stream_executor/cuda/cuda_driver.cc:753] failed to allocate 8.00GiB (8589934592 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2023-11-23 19:28:29.489878: I tensorflow/compiler/xla/stream_executor/cuda/cuda_driver.cc:753] failed to allocate 8.00GiB (8589934592 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2023-11-23 19:28:29.490953: I tensorflow/compiler/xla/stream_executor/cuda/cuda_driver.cc:753] failed to allocate 8.00GiB (8589934592 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2023-11-23 19:28:29.492039: I tensorflow/compiler/xla/stream_executor/cuda/cuda_driver.cc:753] failed to allocate 8.00GiB (8589934592 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2023-11-23 19:28:29.493116: I tensorflow/compiler/xla/stream_executor/cuda/cuda_driver.cc:753] failed to allocate 8.00GiB (8589934592 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2023-11-23 19:28:29.494196: I tensorflow/compiler/xla/stream_executor/cuda/cuda_driver.cc:753] failed to allocate 8.00GiB (8589934592 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2023-11-23 19:28:29.495274: I tensorflow/compiler/xla/stream_executor/cuda/cuda_driver.cc:753] failed to allocate 8.00GiB (8589934592 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2023-11-23 19:28:29.496375: I tensorflow/compiler/xla/stream_executor/cuda/cuda_driver.cc:753] failed to allocate 8.00GiB (8589934592 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2023-11-23 19:28:29.497452: I tensorflow/compiler/xla/stream_executor/cuda/cuda_driver.cc:753] failed to allocate 8.00GiB (8589934592 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2023-11-23 19:28:29.498529: I tensorflow/compiler/xla/stream_executor/cuda/cuda_driver.cc:753] failed to allocate 8.00GiB (8589934592 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2023-11-23 19:28:29.500363: I tensorflow/compiler/xla/stream_executor/cuda/cuda_driver.cc:753] failed to allocate 8.00GiB (8589934592 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2023-11-23 19:28:29.501438: I tensorflow/compiler/xla/stream_executor/cuda/cuda_driver.cc:753] failed to allocate 8.00GiB (8589934592 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2023-11-23 19:28:39.502701: I tensorflow/compiler/xla/stream_executor/cuda/cuda_driver.cc:753] failed to allocate 8.00GiB (8589934592 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2023-11-23 19:28:39.503801: I tensorflow/compiler/xla/stream_executor/cuda/cuda_driver.cc:753] failed to allocate 8.00GiB (8589934592 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2023-11-23 19:28:39.503818: W tensorflow/tsl/framework/bfc_allocator.cc:485] Allocator (GPU_0_bfc) ran out of memory trying to allocate 11.23MiB (rounded to 11780096)requested by op gradient_tape/cnn1d/conv1d_1/Conv1D/Conv2DBackpropFilter-0-TransposeNHWCToNCHW-LayoutOptimizer
If the cause is memory fragmentation maybe the environment variable 'TF_GPU_ALLOCATOR=cuda_malloc_async' will improve the situation.
Current allocation summary follows.
Current allocation summary follows.
2023-11-23 19:28:39.503830: I tensorflow/tsl/framework/bfc_allocator.cc:1039] BFCAllocator dump for GPU_0_bfc
2023-11-23 19:28:39.503839: I tensorflow/tsl/framework/bfc_allocator.cc:1046] Bin (256): Total Chunks: 120, Chunks in use: 119. 30.0KiB allocated for chunks. 29.8KiB in use in bin. 5.1KiB client-requested in use in bin.
2023-11-23 19:28:39.503846: I tensorflow/tsl/framework/bfc_allocator.cc:1046] Bin (512): Total Chunks: 15, Chunks in use: 15. 8.0KiB allocated for chunks. 8.0KiB in use in bin. 7.8KiB client-requested in use in bin.
2023-11-23 19:28:39.503853: I tensorflow/tsl/framework/bfc_allocator.cc:1046] Bin (1024): Total Chunks: 16, Chunks in use: 16. 17.0KiB allocated for chunks. 17.0KiB in use in bin. 15.9KiB client-requested in use in bin.
2023-11-23 19:28:39.503860: I tensorflow/tsl/framework/bfc_allocator.cc:1046] Bin (2048): Total Chunks: 6, Chunks in use: 6. 20.8KiB allocated for chunks. 20.8KiB in use in bin. 18.4KiB client-requested in use in bin.
2023-11-23 19:28:39.503867: I tensorflow/tsl/framework/bfc_allocator.cc:1046] Bin (4096): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2023-11-23 19:28:39.503874: I tensorflow/tsl/framework/bfc_allocator.cc:1046] Bin (8192): Total Chunks: 2, Chunks in use: 2. 24.0KiB allocated for chunks. 24.0KiB in use in bin. 24.0KiB client-requested in use in bin.
2023-11-23 19:28:39.503881: I tensorflow/tsl/framework/bfc_allocator.cc:1046] Bin (16384): Total Chunks: 1, Chunks in use: 1. 17.2KiB allocated for chunks. 17.2KiB in use in bin. 12.0KiB client-requested in use in bin.
2023-11-23 19:28:39.503889: I tensorflow/tsl/framework/bfc_allocator.cc:1046] Bin (32768): Total Chunks: 3, Chunks in use: 3. 115.0KiB allocated for chunks. 115.0KiB in use in bin. 96.0KiB client-requested in use in bin.
2023-11-23 19:28:39.503896: I tensorflow/tsl/framework/bfc_allocator.cc:1046] Bin (65536): Total Chunks: 3, Chunks in use: 3. 242.5KiB allocated for chunks. 242.5KiB in use in bin. 240.0KiB client-requested in use in bin.
2023-11-23 19:28:39.503901: I tensorflow/tsl/framework/bfc_allocator.cc:1046] Bin (131072): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2023-11-23 19:28:39.503913: I tensorflow/tsl/framework/bfc_allocator.cc:1046] Bin (262144): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2023-11-23 19:28:39.503920: I tensorflow/tsl/framework/bfc_allocator.cc:1046] Bin (524288): Total Chunks: 1, Chunks in use: 1. 526.8KiB allocated for chunks. 526.8KiB in use in bin. 384.0KiB client-requested in use in bin.
2023-11-23 19:28:39.503927: I tensorflow/tsl/framework/bfc_allocator.cc:1046] Bin (1048576): Total Chunks: 3, Chunks in use: 1. 3.48MiB allocated for chunks. 1.23MiB in use in bin. 1.23MiB client-requested in use in bin.
2023-11-23 19:28:39.503933: I tensorflow/tsl/framework/bfc_allocator.cc:1046] Bin (2097152): Total Chunks: 1, Chunks in use: 1. 3.69MiB allocated for chunks. 3.69MiB in use in bin. 3.69MiB client-requested in use in bin.
2023-11-23 19:28:39.503939: I tensorflow/tsl/framework/bfc_allocator.cc:1046] Bin (4194304): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2023-11-23 19:28:39.503946: I tensorflow/tsl/framework/bfc_allocator.cc:1046] Bin (8388608): Total Chunks: 2, Chunks in use: 1. 21.08MiB allocated for chunks. 11.23MiB in use in bin. 11.23MiB client-requested in use in bin.
2023-11-23 19:28:39.503953: I tensorflow/tsl/framework/bfc_allocator.cc:1046] Bin (16777216): Total Chunks: 3, Chunks in use: 3. 59.27MiB allocated for chunks. 59.27MiB in use in bin. 43.04MiB client-requested in use in bin.
2023-11-23 19:28:39.503960: I tensorflow/tsl/framework/bfc_allocator.cc:1046] Bin (33554432): Total Chunks: 1, Chunks in use: 1. 41.50MiB allocated for chunks. 41.50MiB in use in bin. 22.50MiB client-requested in use in bin.
2023-11-23 19:28:39.503965: I tensorflow/tsl/framework/bfc_allocator.cc:1046] Bin (67108864): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2023-11-23 19:28:39.503971: I tensorflow/tsl/framework/bfc_allocator.cc:1046] Bin (134217728): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2023-11-23 19:28:39.503983: I tensorflow/tsl/framework/bfc_allocator.cc:1046] Bin (268435456): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2023-11-23 19:28:39.503989: I tensorflow/tsl/framework/bfc_allocator.cc:1062] Bin for 11.23MiB was 8.00MiB, Chunk State:
2023-11-23 19:28:39.503998: I tensorflow/tsl/framework/bfc_allocator.cc:1068] Size: 9.85MiB | Requested Size: 512B | in_use: 0 | bin_num: 15, prev: Size: 1.23MiB | Requested Size: 1.23MiB | in_use: 1 | bin_num: -1
2023-11-23 19:28:39.504003: I tensorflow/tsl/framework/bfc_allocator.cc:1075] Next region of size 67108864
2023-11-23 19:28:39.504009: I tensorflow/tsl/framework/bfc_allocator.cc:1095] InUse at 14962a000000 of size 23592960 next 91
2023-11-23 19:28:39.504015: I tensorflow/tsl/framework/bfc_allocator.cc:1095] InUse at 14962b680000 of size 43515904 next 18446744073709551615
2023-11-23 19:28:39.504020: I tensorflow/tsl/framework/bfc_allocator.cc:1075] Next region of size 33554432
2023-11-23 19:28:39.504025: I tensorflow/tsl/framework/bfc_allocator.cc:1095] InUse at 14962e000000 of size 11780096 next 94
2023-11-23 19:28:39.504031: I tensorflow/tsl/framework/bfc_allocator.cc:1095] InUse at 14962eb3c000 of size 21774336 next 18446744073709551615
2023-11-23 19:28:39.504036: I tensorflow/tsl/framework/bfc_allocator.cc:1075] Next region of size 16777216
2023-11-23 19:28:39.504041: I tensorflow/tsl/framework/bfc_allocator.cc:1095] InUse at 1496ae000000 of size 16777216 next 18446744073709551615
2023-11-23 19:28:39.504046: I tensorflow/tsl/framework/bfc_allocator.cc:1075] Next region of size 2097152
2023-11-23 19:28:39.504051: I tensorflow/tsl/framework/bfc_allocator.cc:1095] InUse at 149724e00000 of size 256 next 1
2023-11-23 19:28:39.504056: I tensorflow/tsl/framework/bfc_allocator.cc:1095] InUse at 149724e00100 of size 1280 next 2
2023-11-23 19:28:39.504064: I tensorflow/tsl/framework/bfc_allocator.cc:1095] InUse at 149724e00600 of size 256 next 3
2023-11-23 19:28:39.504069: I tensorflow/tsl/framework/bfc_allocator.cc:1095] InUse at 149724e00700 of size 256 next 4
2023-11-23 19:28:39.504074: I tensorflow/tsl/framework/bfc_allocator.cc:1095] InUse at 149724e00800 of size 256 next 11
2023-11-23 19:28:39.504078: I tensorflow/tsl/framework/bfc_allocator.cc:1095] InUse at 149724e00900 of size 256 next 6
2023-11-23 19:28:39.504083: I tensorflow/tsl/framework/bfc_allocator.cc:1095] InUse at 149724e00a00 of size 256 next 7
2023-11-23 19:28:39.504088: I tensorflow/tsl/framework/bfc_allocator.cc:1095] InUse at 149724e00b00 of size 256 next 14
2023-11-23 19:28:39.504092: I tensorflow/tsl/framework/bfc_allocator.cc:1095] InUse at 149724e00c00 of size 256 next 10
2023-11-23 19:28:39.504097: I tensorflow/tsl/framework/bfc_allocator.cc:1095] InUse at 149724e00d00 of size 256 next 29
2023-11-23 19:28:39.504102: I tensorflow/tsl/framework/bfc_allocator.cc:1095] InUse at 149724e00e00 of size 256 next 133
2023-11-23 19:28:39.504107: I tensorflow/tsl/framework/bfc_allocator.cc:1095] InUse at 149724e00f00 of size 256 next 33
2023-11-23 19:28:39.504112: I tensorflow/tsl/framework/bfc_allocator.cc:1095] InUse at 149724e01000 of size 256 next 34
2023-11-23 19:28:39.504116: I tensorflow/tsl/framework/bfc_allocator.cc:1095] InUse at 149724e01100 of size 256 next 38
2023-11-23 19:28:39.504121: I tensorflow/tsl/framework/bfc_allocator.cc:1095] InUse at 149724e01200 of size 256 next 15
2023-11-23 19:28:39.504126: I tensorflow/tsl/framework/bfc_allocator.cc:1095] InUse at 149724e01300 of size 256 next 13
2023-11-23 19:28:39.504131: I tensorflow/tsl/framework/bfc_allocator.cc:1095] InUse at 149724e01400 of size 768 next 17
2023-11-23 19:28:39.504135: I tensorflow/tsl/framework/bfc_allocator.cc:1095] InUse at 149724e01700 of size 256 next 18
2023-11-23 19:28:39.504140: I tensorflow/tsl/framework/bfc_allocator.cc:1095] InUse at 149724e01800 of size 256 next 19
2023-11-23 19:28:39.504145: I tensorflow/tsl/framework/bfc_allocator.cc:1095] InUse at 149724e01900 of size 256 next 20
2023-11-23 19:28:39.504150: I tensorflow/tsl/framework/bfc_allocator.cc:1095] InUse at 149724e01a00 of size 256 next 21
2023-11-23 19:28:39.504155: I tensorflow/tsl/framework/bfc_allocator.cc:1095] InUse at 149724e01b00 of size 256 next 22
2023-11-23 19:28:39.504160: I tensorflow/tsl/framework/bfc_allocator.cc:1095] InUse at 149724e01c00 of size 256 next 24
2023-11-23 19:28:39.504164: I tensorflow/tsl/framework/bfc_allocator.cc:1095] InUse at 149724e01d00 of size 256 next 25
2023-11-23 19:28:39.504169: I tensorflow/tsl/framework/bfc_allocator.cc:1095] InUse at 149724e01e00 of size 1024 next 30
2023-11-23 19:28:39.504174: I tensorflow/tsl/framework/bfc_allocator.cc:1095] InUse at 149724e02200 of size 256 next 31
2023-11-23 19:28:39.504179: I tensorflow/tsl/framework/bfc_allocator.cc:1095] InUse at 149724e02300 of size 256 next 32
2023-11-23 19:28:39.504184: I tensorflow/tsl/framework/bfc_allocator.cc:1095] InUse at 149724e02400 of size 512 next 16
2023-11-23 19:28:39.504188: I tensorflow/tsl/framework/bfc_allocator.cc:1095] InUse at 149724e02600 of size 768 next 39
2023-11-23 19:28:39.504193: I tensorflow/tsl/framework/bfc_allocator.cc:1095] InUse at 149724e02900 of size 256 next 40
2023-11-23 19:28:39.504198: I tensorflow/tsl/framework/bfc_allocator.cc:1095] InUse at 149724e02a00 of size 256 next 41
2023-11-23 19:28:39.504203: I tensorflow/tsl/framework/bfc_allocator.cc:1095] InUse at 149724e02b00 of size 1024 next 54
2023-11-23 19:28:39.504208: I tensorflow/tsl/framework/bfc_allocator.cc:1095] InUse at 149724e02f00 of size 1536 next 48
2023-11-23 19:28:39.504213: I tensorflow/tsl/framework/bfc_allocator.cc:1095] InUse at 149724e03500 of size 256 next 49
2023-11-23 19:28:39.504218: I tensorflow/tsl/framework/bfc_allocator.cc:1095] InUse at 149724e03600 of size 256 next 50
2023-11-23 19:28:39.504223: I tensorflow/tsl/framework/bfc_allocator.cc:1095] InUse at 149724e03700 of size 512 next 59
2023-11-23 19:28:39.504231: I tensorflow/tsl/framework/bfc_allocator.cc:1095] InUse at 149724e03900 of size 512 next 56
2023-11-23 19:28:39.504236: I tensorflow/tsl/framework/bfc_allocator.cc:1095] InUse at 149724e03b00 of size 512 next 57
2023-11-23 19:28:39.504241: I tensorflow/tsl/framework/bfc_allocator.cc:1095] InUse at 149724e03d00 of size 512 next 53
2023-11-23 19:28:39.504246: I tensorflow/tsl/framework/bfc_allocator.cc:1095] InUse at 149724e03f00 of size 1024 next 55
2023-11-23 19:28:39.504251: I tensorflow/tsl/framework/bfc_allocator.cc:1095] InUse at 149724e04300 of size 3584 next 44
2023-11-23 19:28:39.504256: I tensorflow/tsl/framework/bfc_allocator.cc:1095] InUse at 149724e05100 of size 256 next 46
2023-11-23 19:28:39.504262: I tensorflow/tsl/framework/bfc_allocator.cc:1095] InUse at 149724e05200 of size 256 next 45
2023-11-23 19:28:39.504267: I tensorflow/tsl/framework/bfc_allocator.cc:1095] InUse at 149724e05300 of size 256 next 35
2023-11-23 19:28:39.504272: I tensorflow/tsl/framework/bfc_allocator.cc:1095] InUse at 149724e05400 of size 1280 next 62
2023-11-23 19:28:39.504277: I tensorflow/tsl/framework/bfc_allocator.cc:1095] InUse at 149724e05900 of size 256 next 63
2023-11-23 19:28:39.504281: I tensorflow/tsl/framework/bfc_allocator.cc:1095] InUse at 149724e05a00 of size 256 next 64
2023-11-23 19:28:39.504286: I tensorflow/tsl/framework/bfc_allocator.cc:1095] InUse at 149724e05b00 of size 256 next 66
2023-11-23 19:28:39.504291: I tensorflow/tsl/framework/bfc_allocator.cc:1095] InUse at 149724e05c00 of size 256 next 67
2023-11-23 19:28:39.504296: I tensorflow/tsl/framework/bfc_allocator.cc:1095] InUse at 149724e05d00 of size 32768 next 27
2023-11-23 19:28:39.504301: I tensorflow/tsl/framework/bfc_allocator.cc:1095] InUse at 149724e0dd00 of size 3584 next 43
2023-11-23 19:28:39.504306: I tensorflow/tsl/framework/bfc_allocator.cc:1095] InUse at 149724e0eb00 of size 256 next 61
2023-11-23 19:28:39.504310: I tensorflow/tsl/framework/bfc_allocator.cc:1095] InUse at 149724e0ec00 of size 256 next 143
2023-11-23 19:28:39.504315: I tensorflow/tsl/framework/bfc_allocator.cc:1095] InUse at 149724e0ed00 of size 256 next 58
2023-11-23 19:28:39.504320: I tensorflow/tsl/framework/bfc_allocator.cc:1095] InUse at 149724e0ee00 of size 256 next 193
2023-11-23 19:28:39.504325: I tensorflow/tsl/framework/bfc_allocator.cc:1095] InUse at 149724e0ef00 of size 256 next 69
2023-11-23 19:28:39.504330: I tensorflow/tsl/framework/bfc_allocator.cc:1095] InUse at 149724e0f000 of size 256 next 70
2023-11-23 19:28:39.504334: I tensorflow/tsl/framework/bfc_allocator.cc:1095] InUse at 149724e0f100 of size 52224 next 106
2023-11-23 19:28:39.504340: I tensorflow/tsl/framework/bfc_allocator.cc:1095] InUse at 149724e1bd00 of size 256 next 120
2023-11-23 19:28:39.504345: I tensorflow/tsl/framework/bfc_allocator.cc:1095] InUse at 149724e1be00 of size 256 next 124
2023-11-23 19:28:39.504349: I tensorflow/tsl/framework/bfc_allocator.cc:1095] InUse at 149724e1bf00 of size 256 next 132
2023-11-23 19:28:39.504354: I tensorflow/tsl/framework/bfc_allocator.cc:1095] InUse at 149724e1c000 of size 256 next 121
2023-11-23 19:28:39.504359: I tensorflow/tsl/framework/bfc_allocator.cc:1095] InUse at 149724e1c100 of size 256 next 128
2023-11-23 19:28:39.504364: I tensorflow/tsl/framework/bfc_allocator.cc:1095] InUse at 149724e1c200 of size 256 next 8
2023-11-23 19:28:39.504368: I tensorflow/tsl/framework/bfc_allocator.cc:1095] InUse at 149724e1c300 of size 256 next 131
2023-11-23 19:28:39.504373: I tensorflow/tsl/framework/bfc_allocator.cc:1095] InUse at 149724e1c400 of size 256 next 129
2023-11-23 19:28:39.504378: I tensorflow/tsl/framework/bfc_allocator.cc:1095] InUse at 149724e1c500 of size 256 next 130
2023-11-23 19:28:39.504383: I tensorflow/tsl/framework/bfc_allocator.cc:1095] InUse at 149724e1c600 of size 256 next 115
2023-11-23 19:28:39.504388: I tensorflow/tsl/framework/bfc_allocator.cc:1095] InUse at 149724e1c700 of size 256 next 210
2023-11-23 19:28:39.504392: I tensorflow/tsl/framework/bfc_allocator.cc:1095] InUse at 149724e1c800 of size 3840 next 169
2023-11-23 19:28:39.504400: I tensorflow/tsl/framework/bfc_allocator.cc:1095] InUse at 149724e1d700 of size 512 next 176
2023-11-23 19:28:39.504405: I tensorflow/tsl/framework/bfc_allocator.cc:1095] InUse at 149724e1d900 of size 256 next 207
2023-11-23 19:28:39.504409: I tensorflow/tsl/framework/bfc_allocator.cc:1095] InUse at 149724e1da00 of size 256 next 213
2023-11-23 19:28:39.504414: I tensorflow/tsl/framework/bfc_allocator.cc:1095] InUse at 149724e1db00 of size 3584 next 37
2023-11-23 19:28:39.504419: I tensorflow/tsl/framework/bfc_allocator.cc:1095] InUse at 149724e1e900 of size 84480 next 109
2023-11-23 19:28:39.504424: I tensorflow/tsl/framework/bfc_allocator.cc:1095] InUse at 149724e33300 of size 256 next 174
2023-11-23 19:28:39.504429: I tensorflow/tsl/framework/bfc_allocator.cc:1095] InUse at 149724e33400 of size 256 next 9
2023-11-23 19:28:39.504434: I tensorflow/tsl/framework/bfc_allocator.cc:1095] InUse at 149724e33500 of size 256 next 114
2023-11-23 19:28:39.504439: I tensorflow/tsl/framework/bfc_allocator.cc:1095] InUse at 149724e33600 of size 256 next 116
2023-11-23 19:28:39.504444: I tensorflow/tsl/framework/bfc_allocator.cc:1095] InUse at 149724e33700 of size 256 next 119
2023-11-23 19:28:39.504448: I tensorflow/tsl/framework/bfc_allocator.cc:1095] InUse at 149724e33800 of size 256 next 118
2023-11-23 19:28:39.504453: I tensorflow/tsl/framework/bfc_allocator.cc:1095] InUse at 149724e33900 of size 256 next 127
2023-11-23 19:28:39.504458: I tensorflow/tsl/framework/bfc_allocator.cc:1095] InUse at 149724e33a00 of size 256 next 126
2023-11-23 19:28:39.504463: I tensorflow/tsl/framework/bfc_allocator.cc:1095] InUse at 149724e33b00 of size 256 next 125
2023-11-23 19:28:39.504468: I tensorflow/tsl/framework/bfc_allocator.cc:1095] InUse at 149724e33c00 of size 3840 next 149
2023-11-23 19:28:39.504472: I tensorflow/tsl/framework/bfc_allocator.cc:1095] InUse at 149724e34b00 of size 512 next 138
2023-11-23 19:28:39.504477: I tensorflow/tsl/framework/bfc_allocator.cc:1095] InUse at 149724e34d00 of size 256 next 158
2023-11-23 19:28:39.504482: I tensorflow/tsl/framework/bfc_allocator.cc:1095] InUse at 149724e34e00 of size 256 next 159
2023-11-23 19:28:39.504487: I tensorflow/tsl/framework/bfc_allocator.cc:1095] InUse at 149724e34f00 of size 256 next 147
2023-11-23 19:28:39.504491: I tensorflow/tsl/framework/bfc_allocator.cc:1095] InUse at 149724e35000 of size 256 next 152
2023-11-23 19:28:39.504496: I tensorflow/tsl/framework/bfc_allocator.cc:1095] InUse at 149724e35100 of size 256 next 156
2023-11-23 19:28:39.504501: I tensorflow/tsl/framework/bfc_allocator.cc:1095] InUse at 149724e35200 of size 12288 next 113
2023-11-23 19:28:39.504506: I tensorflow/tsl/framework/bfc_allocator.cc:1095] InUse at 149724e38200 of size 2816 next 123
2023-11-23 19:28:39.504511: I tensorflow/tsl/framework/bfc_allocator.cc:1095] InUse at 149724e38d00 of size 17664 next 177
2023-11-23 19:28:39.504517: I tensorflow/tsl/framework/bfc_allocator.cc:1095] InUse at 149724e3d200 of size 512 next 136
2023-11-23 19:28:39.504521: I tensorflow/tsl/framework/bfc_allocator.cc:1095] InUse at 149724e3d400 of size 256 next 141
2023-11-23 19:28:39.504526: I tensorflow/tsl/framework/bfc_allocator.cc:1095] InUse at 149724e3d500 of size 256 next 157
2023-11-23 19:28:39.504531: I tensorflow/tsl/framework/bfc_allocator.cc:1095] InUse at 149724e3d600 of size 256 next 171
2023-11-23 19:28:39.504536: I tensorflow/tsl/framework/bfc_allocator.cc:1095] InUse at 149724e3d700 of size 256 next 165
2023-11-23 19:28:39.504541: I tensorflow/tsl/framework/bfc_allocator.cc:1095] InUse at 149724e3d800 of size 12288 next 117
2023-11-23 19:28:39.504545: I tensorflow/tsl/framework/bfc_allocator.cc:1095] InUse at 149724e40800 of size 256 next 112
2023-11-23 19:28:39.504550: I tensorflow/tsl/framework/bfc_allocator.cc:1095] InUse at 149724e40900 of size 256 next 111
2023-11-23 19:28:39.504555: I tensorflow/tsl/framework/bfc_allocator.cc:1095] InUse at 149724e40a00 of size 256 next 110
2023-11-23 19:28:39.504560: I tensorflow/tsl/framework/bfc_allocator.cc:1095] InUse at 149724e40b00 of size 256 next 160
2023-11-23 19:28:39.504566: I tensorflow/tsl/framework/bfc_allocator.cc:1095] InUse at 149724e40c00 of size 256 next 154
2023-11-23 19:28:39.504571: I tensorflow/tsl/framework/bfc_allocator.cc:1095] InUse at 149724e40d00 of size 256 next 178
2023-11-23 19:28:39.504576: I tensorflow/tsl/framework/bfc_allocator.cc:1095] InUse at 149724e40e00 of size 256 next 134
2023-11-23 19:28:39.504581: I tensorflow/tsl/framework/bfc_allocator.cc:1095] InUse at 149724e40f00 of size 256 next 179
2023-11-23 19:28:39.504585: I tensorflow/tsl/framework/bfc_allocator.cc:1095] InUse at 149724e41000 of size 1024 next 99
2023-11-23 19:28:39.504590: I tensorflow/tsl/framework/bfc_allocator.cc:1095] InUse at 149724e41400 of size 1024 next 42
2023-11-23 19:28:39.504595: I tensorflow/tsl/framework/bfc_allocator.cc:1095] InUse at 149724e41800 of size 256 next 47
2023-11-23 19:28:39.504600: I tensorflow/tsl/framework/bfc_allocator.cc:1095] InUse at 149724e41900 of size 256 next 51
2023-11-23 19:28:39.504605: I tensorflow/tsl/framework/bfc_allocator.cc:1095] InUse at 149724e41a00 of size 256 next 52
2023-11-23 19:28:39.504609: I tensorflow/tsl/framework/bfc_allocator.cc:1095] InUse at 149724e41b00 of size 512 next 28
2023-11-23 19:28:39.504614: I tensorflow/tsl/framework/bfc_allocator.cc:1095] InUse at 149724e41d00 of size 512 next 201
2023-11-23 19:28:39.504619: I tensorflow/tsl/framework/bfc_allocator.cc:1095] InUse at 149724e41f00 of size 256 next 163
2023-11-23 19:28:39.504624: I tensorflow/tsl/framework/bfc_allocator.cc:1095] InUse at 149724e42000 of size 256 next 135
2023-11-23 19:28:39.504629: I tensorflow/tsl/framework/bfc_allocator.cc:1095] InUse at 149724e42100 of size 256 next 144
2023-11-23 19:28:39.504634: I tensorflow/tsl/framework/bfc_allocator.cc:1095] InUse at 149724e42200 of size 256 next 229
2023-11-23 19:28:39.504638: I tensorflow/tsl/framework/bfc_allocator.cc:1095] InUse at 149724e42300 of size 256 next 184
2023-11-23 19:28:39.504643: I tensorflow/tsl/framework/bfc_allocator.cc:1095] InUse at 149724e42400 of size 256 next 220
2023-11-23 19:28:39.504648: I tensorflow/tsl/framework/bfc_allocator.cc:1095] InUse at 149724e42500 of size 256 next 202
2023-11-23 19:28:39.504653: I tensorflow/tsl/framework/bfc_allocator.cc:1095] InUse at 149724e42600 of size 256 next 222
2023-11-23 19:28:39.504657: I tensorflow/tsl/framework/bfc_allocator.cc:1095] InUse at 149724e42700 of size 256 next 89
2023-11-23 19:28:39.504662: I tensorflow/tsl/framework/bfc_allocator.cc:1095] InUse at 149724e42800 of size 512 next 150
2023-11-23 19:28:39.504667: I tensorflow/tsl/framework/bfc_allocator.cc:1095] InUse at 149724e42a00 of size 512 next 92
2023-11-23 19:28:39.504672: I tensorflow/tsl/framework/bfc_allocator.cc:1095] InUse at 149724e42c00 of size 512 next 93
2023-11-23 19:28:39.504677: I tensorflow/tsl/framework/bfc_allocator.cc:1095] Free at 149724e42e00 of size 1071872 next 122
2023-11-23 19:28:39.504681: I tensorflow/tsl/framework/bfc_allocator.cc:1095] InUse at 149724f48900 of size 256 next 137
2023-11-23 19:28:39.504686: I tensorflow/tsl/framework/bfc_allocator.cc:1095] InUse at 149724f48a00 of size 256 next 153
2023-11-23 19:28:39.504691: I tensorflow/tsl/framework/bfc_allocator.cc:1095] InUse at 149724f48b00 of size 256 next 175
2023-11-23 19:28:39.504696: I tensorflow/tsl/framework/bfc_allocator.cc:1095] InUse at 149724f48c00 of size 256 next 183
2023-11-23 19:28:39.504701: I tensorflow/tsl/framework/bfc_allocator.cc:1095] InUse at 149724f48d00 of size 256 next 187
2023-11-23 19:28:39.504705: I tensorflow/tsl/framework/bfc_allocator.cc:1095] InUse at 149724f48e00 of size 256 next 190
2023-11-23 19:28:39.504710: I tensorflow/tsl/framework/bfc_allocator.cc:1095] InUse at 149724f48f00 of size 256 next 188
2023-11-23 19:28:39.504715: I tensorflow/tsl/framework/bfc_allocator.cc:1095] InUse at 149724f49000 of size 256 next 173
2023-11-23 19:28:39.504720: I tensorflow/tsl/framework/bfc_allocator.cc:1095] InUse at 149724f49100 of size 256 next 189
2023-11-23 19:28:39.504724: I tensorflow/tsl/framework/bfc_allocator.cc:1095] InUse at 149724f49200 of size 256 next 191
2023-11-23 19:28:39.504731: I tensorflow/tsl/framework/bfc_allocator.cc:1095] InUse at 149724f49300 of size 81920 next 172
2023-11-23 19:28:39.504737: I tensorflow/tsl/framework/bfc_allocator.cc:1095] InUse at 149724f5d300 of size 81920 next 60
2023-11-23 19:28:39.504741: I tensorflow/tsl/framework/bfc_allocator.cc:1095] InUse at 149724f71300 of size 32768 next 215
2023-11-23 19:28:39.504746: I tensorflow/tsl/framework/bfc_allocator.cc:1095] InUse at 149724f79300 of size 1024 next 206
2023-11-23 19:28:39.504751: I tensorflow/tsl/framework/bfc_allocator.cc:1095] InUse at 149724f79700 of size 1024 next 218
2023-11-23 19:28:39.504756: I tensorflow/tsl/framework/bfc_allocator.cc:1095] InUse at 149724f79b00 of size 1024 next 234
2023-11-23 19:28:39.504761: I tensorflow/tsl/framework/bfc_allocator.cc:1095] InUse at 149724f79f00 of size 1024 next 205
2023-11-23 19:28:39.504765: I tensorflow/tsl/framework/bfc_allocator.cc:1095] InUse at 149724f7a300 of size 1024 next 26
2023-11-23 19:28:39.504770: I tensorflow/tsl/framework/bfc_allocator.cc:1095] InUse at 149724f7a700 of size 1024 next 36
2023-11-23 19:28:39.504775: I tensorflow/tsl/framework/bfc_allocator.cc:1095] InUse at 149724f7ab00 of size 1024 next 23
2023-11-23 19:28:39.504780: I tensorflow/tsl/framework/bfc_allocator.cc:1095] InUse at 149724f7af00 of size 1024 next 65
2023-11-23 19:28:39.504785: I tensorflow/tsl/framework/bfc_allocator.cc:1095] InUse at 149724f7b300 of size 256 next 68
2023-11-23 19:28:39.504790: I tensorflow/tsl/framework/bfc_allocator.cc:1095] InUse at 149724f7b400 of size 256 next 71
2023-11-23 19:28:39.504794: I tensorflow/tsl/framework/bfc_allocator.cc:1095] InUse at 149724f7b500 of size 256 next 72
2023-11-23 19:28:39.504799: I tensorflow/tsl/framework/bfc_allocator.cc:1095] InUse at 149724f7b600 of size 256 next 73
2023-11-23 19:28:39.504804: I tensorflow/tsl/framework/bfc_allocator.cc:1095] InUse at 149724f7b700 of size 256 next 74
2023-11-23 19:28:39.504809: I tensorflow/tsl/framework/bfc_allocator.cc:1095] InUse at 149724f7b800 of size 256 next 75
2023-11-23 19:28:39.504814: I tensorflow/tsl/framework/bfc_allocator.cc:1095] InUse at 149724f7b900 of size 256 next 76
2023-11-23 19:28:39.504818: I tensorflow/tsl/framework/bfc_allocator.cc:1095] InUse at 149724f7ba00 of size 256 next 77
2023-11-23 19:28:39.504823: I tensorflow/tsl/framework/bfc_allocator.cc:1095] InUse at 149724f7bb00 of size 256 next 78
2023-11-23 19:28:39.504828: I tensorflow/tsl/framework/bfc_allocator.cc:1095] InUse at 149724f7bc00 of size 256 next 79
2023-11-23 19:28:39.504832: I tensorflow/tsl/framework/bfc_allocator.cc:1095] InUse at 149724f7bd00 of size 256 next 80
2023-11-23 19:28:39.504837: I tensorflow/tsl/framework/bfc_allocator.cc:1095] InUse at 149724f7be00 of size 256 next 81
2023-11-23 19:28:39.504842: I tensorflow/tsl/framework/bfc_allocator.cc:1095] InUse at 149724f7bf00 of size 256 next 82
2023-11-23 19:28:39.504847: I tensorflow/tsl/framework/bfc_allocator.cc:1095] InUse at 149724f7c000 of size 256 next 83
2023-11-23 19:28:39.504851: I tensorflow/tsl/framework/bfc_allocator.cc:1095] InUse at 149724f7c100 of size 256 next 84
2023-11-23 19:28:39.504856: I tensorflow/tsl/framework/bfc_allocator.cc:1095] InUse at 149724f7c200 of size 256 next 85
2023-11-23 19:28:39.504861: I tensorflow/tsl/framework/bfc_allocator.cc:1095] InUse at 149724f7c300 of size 256 next 86
2023-11-23 19:28:39.504866: I tensorflow/tsl/framework/bfc_allocator.cc:1095] Free at 149724f7c400 of size 256 next 87
2023-11-23 19:28:39.504871: I tensorflow/tsl/framework/bfc_allocator.cc:1095] InUse at 149724f7c500 of size 539392 next 18446744073709551615
2023-11-23 19:28:39.504875: I tensorflow/tsl/framework/bfc_allocator.cc:1075] Next region of size 16777216
2023-11-23 19:28:39.504880: I tensorflow/tsl/framework/bfc_allocator.cc:1095] InUse at 149725000000 of size 3870720 next 166
2023-11-23 19:28:39.504885: I tensorflow/tsl/framework/bfc_allocator.cc:1095] Free at 1497253b1000 of size 1290240 next 88
2023-11-23 19:28:39.504891: I tensorflow/tsl/framework/bfc_allocator.cc:1095] InUse at 1497254ec000 of size 1290240 next 90
2023-11-23 19:28:39.504897: I tensorflow/tsl/framework/bfc_allocator.cc:1095] Free at 149725627000 of size 10326016 next 18446744073709551615
2023-11-23 19:28:39.504902: I tensorflow/tsl/framework/bfc_allocator.cc:1100] Summary of in-use Chunks by size:
2023-11-23 19:28:39.504910: I tensorflow/tsl/framework/bfc_allocator.cc:1103] 119 Chunks of size 256 totalling 29.8KiB
2023-11-23 19:28:39.504916: I tensorflow/tsl/framework/bfc_allocator.cc:1103] 13 Chunks of size 512 totalling 6.5KiB
2023-11-23 19:28:39.504921: I tensorflow/tsl/framework/bfc_allocator.cc:1103] 2 Chunks of size 768 totalling 1.5KiB
2023-11-23 19:28:39.504927: I tensorflow/tsl/framework/bfc_allocator.cc:1103] 13 Chunks of size 1024 totalling 13.0KiB
2023-11-23 19:28:39.504932: I tensorflow/tsl/framework/bfc_allocator.cc:1103] 2 Chunks of size 1280 totalling 2.5KiB
2023-11-23 19:28:39.504938: I tensorflow/tsl/framework/bfc_allocator.cc:1103] 1 Chunks of size 1536 totalling 1.5KiB
2023-11-23 19:28:39.504943: I tensorflow/tsl/framework/bfc_allocator.cc:1103] 1 Chunks of size 2816 totalling 2.8KiB
2023-11-23 19:28:39.504949: I tensorflow/tsl/framework/bfc_allocator.cc:1103] 3 Chunks of size 3584 totalling 10.5KiB
2023-11-23 19:28:39.504954: I tensorflow/tsl/framework/bfc_allocator.cc:1103] 2 Chunks of size 3840 totalling 7.5KiB
2023-11-23 19:28:39.504960: I tensorflow/tsl/framework/bfc_allocator.cc:1103] 2 Chunks of size 12288 totalling 24.0KiB
2023-11-23 19:28:39.504966: I tensorflow/tsl/framework/bfc_allocator.cc:1103] 1 Chunks of size 17664 totalling 17.2KiB
2023-11-23 19:28:39.504971: I tensorflow/tsl/framework/bfc_allocator.cc:1103] 2 Chunks of size 32768 totalling 64.0KiB
2023-11-23 19:28:39.504982: I tensorflow/tsl/framework/bfc_allocator.cc:1103] 1 Chunks of size 52224 totalling 51.0KiB
2023-11-23 19:28:39.504988: I tensorflow/tsl/framework/bfc_allocator.cc:1103] 2 Chunks of size 81920 totalling 160.0KiB
2023-11-23 19:28:39.504994: I tensorflow/tsl/framework/bfc_allocator.cc:1103] 1 Chunks of size 84480 totalling 82.5KiB
2023-11-23 19:28:39.505000: I tensorflow/tsl/framework/bfc_allocator.cc:1103] 1 Chunks of size 539392 totalling 526.8KiB
2023-11-23 19:28:39.505006: I tensorflow/tsl/framework/bfc_allocator.cc:1103] 1 Chunks of size 1290240 totalling 1.23MiB
2023-11-23 19:28:39.505011: I tensorflow/tsl/framework/bfc_allocator.cc:1103] 1 Chunks of size 3870720 totalling 3.69MiB
2023-11-23 19:28:39.505017: I tensorflow/tsl/framework/bfc_allocator.cc:1103] 1 Chunks of size 11780096 totalling 11.23MiB
2023-11-23 19:28:39.505023: I tensorflow/tsl/framework/bfc_allocator.cc:1103] 1 Chunks of size 16777216 totalling 16.00MiB
2023-11-23 19:28:39.505029: I tensorflow/tsl/framework/bfc_allocator.cc:1103] 1 Chunks of size 21774336 totalling 20.77MiB
2023-11-23 19:28:39.505034: I tensorflow/tsl/framework/bfc_allocator.cc:1103] 1 Chunks of size 23592960 totalling 22.50MiB
2023-11-23 19:28:39.505040: I tensorflow/tsl/framework/bfc_allocator.cc:1103] 1 Chunks of size 43515904 totalling 41.50MiB
2023-11-23 19:28:39.505046: I tensorflow/tsl/framework/bfc_allocator.cc:1107] Sum Total of in-use chunks: 117.90MiB
2023-11-23 19:28:39.505051: I tensorflow/tsl/framework/bfc_allocator.cc:1109] Total bytes in pool: 136314880 memory_limit_: 15312289792 available bytes: 15175974912 curr_region_allocation_bytes_: 8589934592
2023-11-23 19:28:39.505060: I tensorflow/tsl/framework/bfc_allocator.cc:1114] Stats:
Limit: 15312289792
InUse: 123626496
MaxInUse: 2897541120
NumAllocs: 26643
MaxAllocSize: 2743205888
Reserved: 0
PeakReserved: 0
LargestFreeBlock: 0
2023-11-23 19:28:39.505074: W tensorflow/tsl/framework/bfc_allocator.cc:497] ***********************************xxxxxxxxxxxxxx******************xxxxxx*********xxxx*******_______
2023-11-23 19:28:39.505093: W tensorflow/core/framework/op_kernel.cc:1828] OP_REQUIRES failed at transpose_op.cc:184 : RESOURCE_EXHAUSTED: OOM when allocating tensor with shape[32,128,1,719] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc
Traceback (most recent call last):
File "/home/rqchia/.local/lib/python3.8/site-packages/keras_tuner/src/engine/base_tuner.py", line 270, in _try_run_and_update_trial
self._run_and_update_trial(trial, *fit_args, **fit_kwargs)
File "/home/rqchia/.local/lib/python3.8/site-packages/keras_tuner/src/engine/base_tuner.py", line 235, in _run_and_update_trial
results = self.run_trial(trial, *fit_args, **fit_kwargs)
File "/home/rqchia/.local/lib/python3.8/site-packages/keras_tuner/src/engine/tuner.py", line 314, in run_trial
obj_value = self._build_and_fit_model(trial, *args, **copied_kwargs)
File "/home/rqchia/.local/lib/python3.8/site-packages/keras_tuner/src/engine/tuner.py", line 233, in _build_and_fit_model
results = self.hypermodel.fit(hp, model, *args, **kwargs)
File "/home/rqchia/projects/aria-respiration-cal/models/neuralnet.py", line 178, in fit
history = model.fit(x, y,
File "/home/rqchia/.local/lib/python3.8/site-packages/keras/src/utils/traceback_utils.py", line 70, in error_handler
raise e.with_traceback(filtered_tb) from None
File "/home/rqchia/.local/lib/python3.8/site-packages/tensorflow/python/eager/execute.py", line 53, in quick_execute
tensors = pywrap_tfe.TFE_Py_Execute(ctx._handle, device_name, op_name,
tensorflow.python.framework.errors_impl.ResourceExhaustedError: Graph execution error:
OOM when allocating tensor with shape[32,128,1,719] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc
[[{{node gradient_tape/cnn1d/conv1d_1/Conv1D/Conv2DBackpropFilter-0-TransposeNHWCToNCHW-LayoutOptimizer}}]]
Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info. This isn't available when running in Eager mode.
[Op:__inference_train_function_22874]
[2K
[2K
Trial 5 Complete [00h 00m 12s]
Best val_loss So Far: 13.035322189331055
Total elapsed time: 00h 00m 51s
Search: Running Trial #6
Value |Best Value So Far |Hyperparameter
3 |5 |n_layers
256 |128 |filter_unit0
256 |128 |filter_unit1
64 |256 |filter_unit2
2 |5 |kernel_size0
1 |5 |kernel_size1
1 |4 |kernel_size2
3 |2 |pool_size0
5 |1 |pool_size1
3 |2 |pool_size2
1 |2 |stride_size0
4 |3 |stride_size1
1 |3 |stride_size2
0.3 |0.1 |dropout0
0.1 |0.3 |dropout1
0.4 |0.4 |dropout2
2023-11-23 19:28:41.149683: I tensorflow/compiler/xla/stream_executor/cuda/cuda_driver.cc:753] failed to allocate 8.00GiB (8589934592 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2023-11-23 19:28:41.149726: W tensorflow/core/kernels/gpu_utils.cc:50] Failed to allocate memory for convolution redzone checking; skipping this check. This is benign and only means that we won't check cudnn for out-of-bounds reads and writes. This message will only be printed once.
2023-11-23 19:28:41.153774: I tensorflow/compiler/xla/stream_executor/cuda/cuda_driver.cc:753] failed to allocate 8.00GiB (8589934592 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2023-11-23 19:28:41.154894: I tensorflow/compiler/xla/stream_executor/cuda/cuda_driver.cc:753] failed to allocate 8.00GiB (8589934592 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2023-11-23 19:28:41.155986: I tensorflow/compiler/xla/stream_executor/cuda/cuda_driver.cc:753] failed to allocate 8.00GiB (8589934592 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2023-11-23 19:28:41.157126: I tensorflow/compiler/xla/stream_executor/cuda/cuda_driver.cc:753] failed to allocate 8.00GiB (8589934592 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2023-11-23 19:28:41.158281: I tensorflow/compiler/xla/stream_executor/cuda/cuda_driver.cc:753] failed to allocate 8.00GiB (8589934592 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2023-11-23 19:28:41.159368: I tensorflow/compiler/xla/stream_executor/cuda/cuda_driver.cc:753] failed to allocate 8.00GiB (8589934592 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2023-11-23 19:28:41.160447: I tensorflow/compiler/xla/stream_executor/cuda/cuda_driver.cc:753] failed to allocate 8.00GiB (8589934592 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2023-11-23 19:28:41.161527: I tensorflow/compiler/xla/stream_executor/cuda/cuda_driver.cc:753] failed to allocate 8.00GiB (8589934592 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2023-11-23 19:28:41.162603: I tensorflow/compiler/xla/stream_executor/cuda/cuda_driver.cc:753] failed to allocate 8.00GiB (8589934592 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2023-11-23 19:28:41.163680: I tensorflow/compiler/xla/stream_executor/cuda/cuda_driver.cc:753] failed to allocate 8.00GiB (8589934592 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2023-11-23 19:28:41.164756: I tensorflow/compiler/xla/stream_executor/cuda/cuda_driver.cc:753] failed to allocate 8.00GiB (8589934592 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2023-11-23 19:28:41.165832: I tensorflow/compiler/xla/stream_executor/cuda/cuda_driver.cc:753] failed to allocate 8.00GiB (8589934592 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2023-11-23 19:28:41.166923: I tensorflow/compiler/xla/stream_executor/cuda/cuda_driver.cc:753] failed to allocate 8.00GiB (8589934592 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2023-11-23 19:28:41.168007: I tensorflow/compiler/xla/stream_executor/cuda/cuda_driver.cc:753] failed to allocate 8.00GiB (8589934592 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2023-11-23 19:28:41.169083: I tensorflow/compiler/xla/stream_executor/cuda/cuda_driver.cc:753] failed to allocate 8.00GiB (8589934592 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2023-11-23 19:28:41.170158: I tensorflow/compiler/xla/stream_executor/cuda/cuda_driver.cc:753] failed to allocate 8.00GiB (8589934592 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2023-11-23 19:28:41.171232: I tensorflow/compiler/xla/stream_executor/cuda/cuda_driver.cc:753] failed to allocate 8.00GiB (8589934592 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2023-11-23 19:28:41.171330: W tensorflow/core/kernels/conv_ops_gpu.cc:321] None of the algorithms provided by cuDNN frontend heuristics worked; trying fallback algorithms. Conv: batch: 32
in_depths: 7
out_depths: 256
in: 1
in: 1441
data_format: 1
filter: 1
filter: 2
filter: 7
dilation: 1
dilation: 1
stride: 1
stride: 1
padding: 0
padding: 0
dtype: DT_FLOAT
group_count: 1
device_identifier: "sm_7.5 with 16900292608B RAM, 48 cores, 1815000KHz clock, 7001000KHz mem clock, 4194304B L2$"
version: 3
2023-11-23 19:28:41.172602: I tensorflow/compiler/xla/stream_executor/cuda/cuda_driver.cc:753] failed to allocate 8.00GiB (8589934592 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2023-11-23 19:28:41.173692: I tensorflow/compiler/xla/stream_executor/cuda/cuda_driver.cc:753] failed to allocate 8.00GiB (8589934592 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2023-11-23 19:28:41.174769: I tensorflow/compiler/xla/stream_executor/cuda/cuda_driver.cc:753] failed to allocate 8.00GiB (8589934592 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2023-11-23 19:28:41.175403: W tensorflow/core/framework/op_kernel.cc:1828] OP_REQUIRES failed at conv_ops_impl.h:770 : NOT_FOUND: No algorithm worked! Error messages:
Profiling failure on CUDNN engine eng1{}: RESOURCE_EXHAUSTED: Out of memory while trying to allocate 16777272 bytes.
Profiling failure on CUDNN engine eng28{}: RESOURCE_EXHAUSTED: Out of memory while trying to allocate 16777216 bytes.
Profiling failure on CUDNN engine eng0{}: RESOURCE_EXHAUSTED: Out of memory while trying to allocate 16777216 bytes.
Traceback (most recent call last):
File "/home/rqchia/.local/lib/python3.8/site-packages/keras_tuner/src/engine/base_tuner.py", line 270, in _try_run_and_update_trial
self._run_and_update_trial(trial, *fit_args, **fit_kwargs)
File "/home/rqchia/.local/lib/python3.8/site-packages/keras_tuner/src/engine/base_tuner.py", line 235, in _run_and_update_trial
results = self.run_trial(trial, *fit_args, **fit_kwargs)
File "/home/rqchia/.local/lib/python3.8/site-packages/keras_tuner/src/engine/tuner.py", line 314, in run_trial
obj_value = self._build_and_fit_model(trial, *args, **copied_kwargs)
File "/home/rqchia/.local/lib/python3.8/site-packages/keras_tuner/src/engine/tuner.py", line 233, in _build_and_fit_model
results = self.hypermodel.fit(hp, model, *args, **kwargs)
File "/home/rqchia/projects/aria-respiration-cal/models/neuralnet.py", line 178, in fit
history = model.fit(x, y,
File "/home/rqchia/.local/lib/python3.8/site-packages/keras/src/utils/traceback_utils.py", line 70, in error_handler
raise e.with_traceback(filtered_tb) from None
File "/home/rqchia/.local/lib/python3.8/site-packages/tensorflow/python/eager/execute.py", line 53, in quick_execute
tensors = pywrap_tfe.TFE_Py_Execute(ctx._handle, device_name, op_name,
tensorflow.python.framework.errors_impl.NotFoundError: Graph execution error:
Detected at node 'cnn1d/conv1d_0/Conv1D' defined at (most recent call last):
File "regress_rr.py", line 1150, in <module>
sens_rr_model(subject,
File "regress_rr.py", line 1034, in sens_rr_model
transforms, model = model_training(mdl_str, x_train, y_train,
File "/home/rqchia/projects/aria-respiration-cal/modules/utils.py", line 260, in model_training
tuner.search(x_train, y_train, validation_data,
File "/home/rqchia/projects/aria-respiration-cal/models/neuralnet.py", line 499, in search
self.tuner.search(x, y, validation_data,
File "/home/rqchia/.local/lib/python3.8/site-packages/keras_tuner/src/engine/base_tuner.py", line 230, in search
self._try_run_and_update_trial(trial, *fit_args, **fit_kwargs)
File "/home/rqchia/.local/lib/python3.8/site-packages/keras_tuner/src/engine/base_tuner.py", line 270, in _try_run_and_update_trial
self._run_and_update_trial(trial, *fit_args, **fit_kwargs)
File "/home/rqchia/.local/lib/python3.8/site-packages/keras_tuner/src/engine/base_tuner.py", line 235, in _run_and_update_trial
results = self.run_trial(trial, *fit_args, **fit_kwargs)
File "/home/rqchia/.local/lib/python3.8/site-packages/keras_tuner/src/engine/tuner.py", line 314, in run_trial
obj_value = self._build_and_fit_model(trial, *args, **copied_kwargs)
File "/home/rqchia/.local/lib/python3.8/site-packages/keras_tuner/src/engine/tuner.py", line 233, in _build_and_fit_model
results = self.hypermodel.fit(hp, model, *args, **kwargs)
File "/home/rqchia/projects/aria-respiration-cal/models/neuralnet.py", line 178, in fit
history = model.fit(x, y,
File "/home/rqchia/.local/lib/python3.8/site-packages/keras/src/utils/traceback_utils.py", line 65, in error_handler
return fn(*args, **kwargs)
File "/home/rqchia/.local/lib/python3.8/site-packages/keras/src/engine/training.py", line 1742, in fit
tmp_logs = self.train_function(iterator)
File "/home/rqchia/.local/lib/python3.8/site-packages/keras/src/engine/training.py", line 1338, in train_function
return step_function(self, iterator)
File "/home/rqchia/.local/lib/python3.8/site-packages/keras/src/engine/training.py", line 1322, in step_function
outputs = model.distribute_strategy.run(run_step, args=(data,))
File "/home/rqchia/.local/lib/python3.8/site-packages/keras/src/engine/training.py", line 1303, in run_step
outputs = model.train_step(data)
File "/home/rqchia/.local/lib/python3.8/site-packages/keras/src/engine/training.py", line 1080, in train_step
y_pred = self(x, training=True)
File "/home/rqchia/.local/lib/python3.8/site-packages/keras/src/utils/traceback_utils.py", line 65, in error_handler
return fn(*args, **kwargs)
File "/home/rqchia/.local/lib/python3.8/site-packages/keras/src/engine/training.py", line 569, in __call__
return super().__call__(*args, **kwargs)
File "/home/rqchia/.local/lib/python3.8/site-packages/keras/src/utils/traceback_utils.py", line 65, in error_handler
return fn(*args, **kwargs)
File "/home/rqchia/.local/lib/python3.8/site-packages/keras/src/engine/base_layer.py", line 1150, in __call__
outputs = call_fn(inputs, *args, **kwargs)
File "/home/rqchia/.local/lib/python3.8/site-packages/keras/src/utils/traceback_utils.py", line 96, in error_handler
return fn(*args, **kwargs)
File "/home/rqchia/.local/lib/python3.8/site-packages/keras/src/engine/sequential.py", line 405, in call
return super().call(inputs, training=training, mask=mask)
File "/home/rqchia/.local/lib/python3.8/site-packages/keras/src/engine/functional.py", line 512, in call
return self._run_internal_graph(inputs, training=training, mask=mask)
File "/home/rqchia/.local/lib/python3.8/site-packages/keras/src/engine/functional.py", line 669, in _run_internal_graph
outputs = node.layer(*args, **kwargs)
File "/home/rqchia/.local/lib/python3.8/site-packages/keras/src/utils/traceback_utils.py", line 65, in error_handler
return fn(*args, **kwargs)
File "/home/rqchia/.local/lib/python3.8/site-packages/keras/src/engine/base_layer.py", line 1150, in __call__
outputs = call_fn(inputs, *args, **kwargs)
File "/home/rqchia/.local/lib/python3.8/site-packages/keras/src/utils/traceback_utils.py", line 96, in error_handler
return fn(*args, **kwargs)
File "/home/rqchia/.local/lib/python3.8/site-packages/keras/src/layers/convolutional/base_conv.py", line 290, in call
outputs = self.convolution_op(inputs, self.kernel)
File "/home/rqchia/.local/lib/python3.8/site-packages/keras/src/layers/convolutional/base_conv.py", line 262, in convolution_op
return tf.nn.convolution(
Node: 'cnn1d/conv1d_0/Conv1D'
No algorithm worked! Error messages:
Profiling failure on CUDNN engine eng1{}: RESOURCE_EXHAUSTED: Out of memory while trying to allocate 16777272 bytes.
Profiling failure on CUDNN engine eng28{}: RESOURCE_EXHAUSTED: Out of memory while trying to allocate 16777216 bytes.
Profiling failure on CUDNN engine eng0{}: RESOURCE_EXHAUSTED: Out of memory while trying to allocate 16777216 bytes.
[[{{node cnn1d/conv1d_0/Conv1D}}]] [Op:__inference_train_function_25315]
[2K
[2K
Trial 6 Complete [00h 00m 03s]
Best val_loss So Far: 13.035322189331055
Total elapsed time: 00h 00m 54s
Search: Running Trial #7
Value |Best Value So Far |Hyperparameter
5 |5 |n_layers
64 |128 |filter_unit0
128 |128 |filter_unit1
128 |256 |filter_unit2
32 |128 |filter_unit3
32 |128 |filter_unit4
4 |5 |kernel_size0
4 |5 |kernel_size1
3 |4 |kernel_size2
2 |2 |kernel_size3
3 |4 |kernel_size4
3 |2 |pool_size0
3 |1 |pool_size1
3 |2 |pool_size2
3 |4 |pool_size3
5 |3 |pool_size4
1 |2 |stride_size0
5 |3 |stride_size1
4 |3 |stride_size2
3 |2 |stride_size3
2 |2 |stride_size4
0.2 |0.1 |dropout0
0.4 |0.3 |dropout1
0.2 |0.4 |dropout2
0.4 |0.4 |dropout3
0.4 |0.2 |dropout4
2023-11-23 19:28:44.646245: I tensorflow/compiler/xla/stream_executor/cuda/cuda_driver.cc:753] failed to allocate 8.00GiB (8589934592 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2023-11-23 19:28:44.647354: I tensorflow/compiler/xla/stream_executor/cuda/cuda_driver.cc:753] failed to allocate 8.00GiB (8589934592 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2023-11-23 19:28:54.648639: I tensorflow/compiler/xla/stream_executor/cuda/cuda_driver.cc:753] failed to allocate 8.00GiB (8589934592 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2023-11-23 19:28:54.649749: I tensorflow/compiler/xla/stream_executor/cuda/cuda_driver.cc:753] failed to allocate 8.00GiB (8589934592 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2023-11-23 19:28:54.649766: W tensorflow/tsl/framework/bfc_allocator.cc:485] Allocator (GPU_0_bfc) ran out of memory trying to allocate 11.24MiB (rounded to 11788288)requested by op cnn1d/conv1d_1/Conv1D
If the cause is memory fragmentation maybe the environment variable 'TF_GPU_ALLOCATOR=cuda_malloc_async' will improve the situation.
Current allocation summary follows.
Current allocation summary follows.
2023-11-23 19:28:54.649780: I tensorflow/tsl/framework/bfc_allocator.cc:1039] BFCAllocator dump for GPU_0_bfc
2023-11-23 19:28:54.649788: I tensorflow/tsl/framework/bfc_allocator.cc:1046] Bin (256): Total Chunks: 146, Chunks in use: 146. 36.5KiB allocated for chunks. 36.5KiB in use in bin. 10.8KiB client-requested in use in bin.
2023-11-23 19:28:54.649795: I tensorflow/tsl/framework/bfc_allocator.cc:1046] Bin (512): Total Chunks: 23, Chunks in use: 23. 12.5KiB allocated for chunks. 12.5KiB in use in bin. 11.8KiB client-requested in use in bin.
2023-11-23 19:28:54.649803: I tensorflow/tsl/framework/bfc_allocator.cc:1046] Bin (1024): Total Chunks: 24, Chunks in use: 24. 25.5KiB allocated for chunks. 25.5KiB in use in bin. 23.9KiB client-requested in use in bin.
2023-11-23 19:28:54.649809: I tensorflow/tsl/framework/bfc_allocator.cc:1046] Bin (2048): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2023-11-23 19:28:54.649815: I tensorflow/tsl/framework/bfc_allocator.cc:1046] Bin (4096): Total Chunks: 2, Chunks in use: 2. 14.0KiB allocated for chunks. 14.0KiB in use in bin. 14.0KiB client-requested in use in bin.
2023-11-23 19:28:54.649822: I tensorflow/tsl/framework/bfc_allocator.cc:1046] Bin (8192): Total Chunks: 6, Chunks in use: 6. 74.0KiB allocated for chunks. 74.0KiB in use in bin. 69.0KiB client-requested in use in bin.
2023-11-23 19:28:54.649829: I tensorflow/tsl/framework/bfc_allocator.cc:1046] Bin (16384): Total Chunks: 3, Chunks in use: 3. 63.0KiB allocated for chunks. 63.0KiB in use in bin. 50.0KiB client-requested in use in bin.
2023-11-23 19:28:54.649836: I tensorflow/tsl/framework/bfc_allocator.cc:1046] Bin (32768): Total Chunks: 3, Chunks in use: 3. 96.0KiB allocated for chunks. 96.0KiB in use in bin. 96.0KiB client-requested in use in bin.
2023-11-23 19:28:54.649843: I tensorflow/tsl/framework/bfc_allocator.cc:1046] Bin (65536): Total Chunks: 3, Chunks in use: 3. 242.0KiB allocated for chunks. 242.0KiB in use in bin. 192.0KiB client-requested in use in bin.
2023-11-23 19:28:54.649849: I tensorflow/tsl/framework/bfc_allocator.cc:1046] Bin (131072): Total Chunks: 6, Chunks in use: 6. 1.03MiB allocated for chunks. 1.03MiB in use in bin. 960.0KiB client-requested in use in bin.
2023-11-23 19:28:54.649856: I tensorflow/tsl/framework/bfc_allocator.cc:1046] Bin (262144): Total Chunks: 4, Chunks in use: 4. 1.11MiB allocated for chunks. 1.11MiB in use in bin. 1.03MiB client-requested in use in bin.
2023-11-23 19:28:54.649862: I tensorflow/tsl/framework/bfc_allocator.cc:1046] Bin (524288): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2023-11-23 19:28:54.649868: I tensorflow/tsl/framework/bfc_allocator.cc:1046] Bin (1048576): Total Chunks: 3, Chunks in use: 2. 3.62MiB allocated for chunks. 2.39MiB in use in bin. 2.36MiB client-requested in use in bin.
2023-11-23 19:28:54.649879: I tensorflow/tsl/framework/bfc_allocator.cc:1046] Bin (2097152): Total Chunks: 2, Chunks in use: 1. 7.11MiB allocated for chunks. 3.69MiB in use in bin. 3.69MiB client-requested in use in bin.
2023-11-23 19:28:54.649886: I tensorflow/tsl/framework/bfc_allocator.cc:1046] Bin (4194304): Total Chunks: 1, Chunks in use: 1. 4.57MiB allocated for chunks. 4.57MiB in use in bin. 2.81MiB client-requested in use in bin.
2023-11-23 19:28:54.649893: I tensorflow/tsl/framework/bfc_allocator.cc:1046] Bin (8388608): Total Chunks: 4, Chunks in use: 3. 44.97MiB allocated for chunks. 33.73MiB in use in bin. 33.72MiB client-requested in use in bin.
2023-11-23 19:28:54.649900: I tensorflow/tsl/framework/bfc_allocator.cc:1046] Bin (16777216): Total Chunks: 3, Chunks in use: 3. 67.03MiB allocated for chunks. 67.03MiB in use in bin. 43.02MiB client-requested in use in bin.
2023-11-23 19:28:54.649906: I tensorflow/tsl/framework/bfc_allocator.cc:1046] Bin (33554432): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2023-11-23 19:28:54.649912: I tensorflow/tsl/framework/bfc_allocator.cc:1046] Bin (67108864): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2023-11-23 19:28:54.649917: I tensorflow/tsl/framework/bfc_allocator.cc:1046] Bin (134217728): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2023-11-23 19:28:54.649922: I tensorflow/tsl/framework/bfc_allocator.cc:1046] Bin (268435456): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2023-11-23 19:28:54.649929: I tensorflow/tsl/framework/bfc_allocator.cc:1062] Bin for 11.24MiB was 8.00MiB, Chunk State:
2023-11-23 19:28:54.649940: I tensorflow/tsl/framework/bfc_allocator.cc:1068] Size: 11.23MiB | Requested Size: 11.23MiB | in_use: 0 | bin_num: 15, prev: Size: 11.25MiB | Requested Size: 11.25MiB | in_use: 1 | bin_num: -1, next: Size: 11.23MiB | Requested Size: 11.23MiB | in_use: 1 | bin_num: -1
2023-11-23 19:28:54.649945: I tensorflow/tsl/framework/bfc_allocator.cc:1075] Next region of size 67108864
2023-11-23 19:28:54.649952: I tensorflow/tsl/framework/bfc_allocator.cc:1095] InUse at 14962a000000 of size 11796480 next 221
2023-11-23 19:28:54.649956: I tensorflow/tsl/framework/bfc_allocator.cc:1095] Free at 14962ab40000 of size 11780096 next 232
2023-11-23 19:28:54.649962: I tensorflow/tsl/framework/bfc_allocator.cc:1095] InUse at 14962b67c000 of size 11780096 next 225
2023-11-23 19:28:54.649967: I tensorflow/tsl/framework/bfc_allocator.cc:1095] InUse at 14962c1b8000 of size 31752192 next 18446744073709551615
2023-11-23 19:28:54.649972: I tensorflow/tsl/framework/bfc_allocator.cc:1075] Next region of size 33554432
2023-11-23 19:28:54.649983: I tensorflow/tsl/framework/bfc_allocator.cc:1095] InUse at 14962e000000 of size 11796480 next 239
2023-11-23 19:28:54.649990: I tensorflow/tsl/framework/bfc_allocator.cc:1095] InUse at 14962eb40000 of size 21757952 next 18446744073709551615
2023-11-23 19:28:54.649994: I tensorflow/tsl/framework/bfc_allocator.cc:1075] Next region of size 16777216
2023-11-23 19:28:54.649999: I tensorflow/tsl/framework/bfc_allocator.cc:1095] InUse at 1496ae000000 of size 16777216 next 18446744073709551615
2023-11-23 19:28:54.650004: I tensorflow/tsl/framework/bfc_allocator.cc:1075] Next region of size 2097152
2023-11-23 19:28:54.650009: I tensorflow/tsl/framework/bfc_allocator.cc:1095] InUse at 149724e00000 of size 256 next 1
2023-11-23 19:28:54.650014: I tensorflow/tsl/framework/bfc_allocator.cc:1095] InUse at 149724e00100 of size 1280 next 2
2023-11-23 19:28:54.650019: I tensorflow/tsl/framework/bfc_allocator.cc:1095] InUse at 149724e00600 of size 256 next 3
2023-11-23 19:28:54.650024: I tensorflow/tsl/framework/bfc_allocator.cc:1095] InUse at 149724e00700 of size 256 next 4
2023-11-23 19:28:54.650032: I tensorflow/tsl/framework/bfc_allocator.cc:1095] InUse at 149724e00800 of size 256 next 11
2023-11-23 19:28:54.650037: I tensorflow/tsl/framework/bfc_allocator.cc:1095] InUse at 149724e00900 of size 256 next 6
2023-11-23 19:28:54.650041: I tensorflow/tsl/framework/bfc_allocator.cc:1095] InUse at 149724e00a00 of size 256 next 7
2023-11-23 19:28:54.650046: I tensorflow/tsl/framework/bfc_allocator.cc:1095] InUse at 149724e00b00 of size 256 next 14
2023-11-23 19:28:54.650051: I tensorflow/tsl/framework/bfc_allocator.cc:1095] InUse at 149724e00c00 of size 256 next 10
2023-11-23 19:28:54.650055: I tensorflow/tsl/framework/bfc_allocator.cc:1095] InUse at 149724e00d00 of size 256 next 27
2023-11-23 19:28:54.650060: I tensorflow/tsl/framework/bfc_allocator.cc:1095] InUse at 149724e00e00 of size 512 next 34
2023-11-23 19:28:54.650065: I tensorflow/tsl/framework/bfc_allocator.cc:1095] InUse at 149724e01000 of size 512 next 38
2023-11-23 19:28:54.650070: I tensorflow/tsl/framework/bfc_allocator.cc:1095] InUse at 149724e01200 of size 256 next 15
2023-11-23 19:28:54.650074: I tensorflow/tsl/framework/bfc_allocator.cc:1095] InUse at 149724e01300 of size 256 next 13
2023-11-23 19:28:54.650079: I tensorflow/tsl/framework/bfc_allocator.cc:1095] InUse at 149724e01400 of size 768 next 17
2023-11-23 19:28:54.650084: I tensorflow/tsl/framework/bfc_allocator.cc:1095] InUse at 149724e01700 of size 256 next 18
2023-11-23 19:28:54.650089: I tensorflow/tsl/framework/bfc_allocator.cc:1095] InUse at 149724e01800 of size 256 next 19
2023-11-23 19:28:54.650093: I tensorflow/tsl/framework/bfc_allocator.cc:1095] InUse at 149724e01900 of size 256 next 20
2023-11-23 19:28:54.650098: I tensorflow/tsl/framework/bfc_allocator.cc:1095] InUse at 149724e01a00 of size 256 next 21
2023-11-23 19:28:54.650103: I tensorflow/tsl/framework/bfc_allocator.cc:1095] InUse at 149724e01b00 of size 256 next 22
2023-11-23 19:28:54.650108: I tensorflow/tsl/framework/bfc_allocator.cc:1095] InUse at 149724e01c00 of size 256 next 24
2023-11-23 19:28:54.650112: I tensorflow/tsl/framework/bfc_allocator.cc:1095] InUse at 149724e01d00 of size 256 next 25
2023-11-23 19:28:54.650117: I tensorflow/tsl/framework/bfc_allocator.cc:1095] InUse at 149724e01e00 of size 1024 next 30
2023-11-23 19:28:54.650122: I tensorflow/tsl/framework/bfc_allocator.cc:1095] InUse at 149724e02200 of size 256 next 31
2023-11-23 19:28:54.650127: I tensorflow/tsl/framework/bfc_allocator.cc:1095] InUse at 149724e02300 of size 256 next 32
2023-11-23 19:28:54.650131: I tensorflow/tsl/framework/bfc_allocator.cc:1095] InUse at 149724e02400 of size 512 next 33
2023-11-23 19:28:54.650136: I tensorflow/tsl/framework/bfc_allocator.cc:1095] InUse at 149724e02600 of size 768 next 39
2023-11-23 19:28:54.650141: I tensorflow/tsl/framework/bfc_allocator.cc:1095] InUse at 149724e02900 of size 256 next 40
2023-11-23 19:28:54.650146: I tensorflow/tsl/framework/bfc_allocator.cc:1095] InUse at 149724e02a00 of size 256 next 41
2023-11-23 19:28:54.650150: I tensorflow/tsl/framework/bfc_allocator.cc:1095] InUse at 149724e02b00 of size 256 next 61
2023-11-23 19:28:54.650154: I tensorflow/tsl/framework/bfc_allocator.cc:1095] InUse at 149724e02c00 of size 256 next 143
2023-11-23 19:28:54.650159: I tensorflow/tsl/framework/bfc_allocator.cc:1095] InUse at 149724e02d00 of size 256 next 58
2023-11-23 19:28:54.650164: I tensorflow/tsl/framework/bfc_allocator.cc:1095] InUse at 149724e02e00 of size 256 next 70
2023-11-23 19:28:54.650169: I tensorflow/tsl/framework/bfc_allocator.cc:1095] InUse at 149724e02f00 of size 256 next 215
2023-11-23 19:28:54.650173: I tensorflow/tsl/framework/bfc_allocator.cc:1095] InUse at 149724e03000 of size 256 next 206
2023-11-23 19:28:54.650178: I tensorflow/tsl/framework/bfc_allocator.cc:1095] InUse at 149724e03100 of size 256 next 234
2023-11-23 19:28:54.650183: I tensorflow/tsl/framework/bfc_allocator.cc:1095] InUse at 149724e03200 of size 256 next 26
2023-11-23 19:28:54.650188: I tensorflow/tsl/framework/bfc_allocator.cc:1095] InUse at 149724e03300 of size 256 next 205
2023-11-23 19:28:54.650194: I tensorflow/tsl/framework/bfc_allocator.cc:1095] InUse at 149724e03400 of size 256 next 48
2023-11-23 19:28:54.650199: I tensorflow/tsl/framework/bfc_allocator.cc:1095] InUse at 149724e03500 of size 256 next 49
2023-11-23 19:28:54.650204: I tensorflow/tsl/framework/bfc_allocator.cc:1095] InUse at 149724e03600 of size 256 next 50
2023-11-23 19:28:54.650209: I tensorflow/tsl/framework/bfc_allocator.cc:1095] InUse at 149724e03700 of size 256 next 36
2023-11-23 19:28:54.650213: I tensorflow/tsl/framework/bfc_allocator.cc:1095] InUse at 149724e03800 of size 256 next 23
2023-11-23 19:28:54.650218: I tensorflow/tsl/framework/bfc_allocator.cc:1095] InUse at 149724e03900 of size 256 next 65
2023-11-23 19:28:54.650223: I tensorflow/tsl/framework/bfc_allocator.cc:1095] InUse at 149724e03a00 of size 256 next 68
2023-11-23 19:28:54.650228: I tensorflow/tsl/framework/bfc_allocator.cc:1095] InUse at 149724e03b00 of size 512 next 104
2023-11-23 19:28:54.650232: I tensorflow/tsl/framework/bfc_allocator.cc:1095] InUse at 149724e03d00 of size 512 next 105
2023-11-23 19:28:54.650237: I tensorflow/tsl/framework/bfc_allocator.cc:1095] InUse at 149724e03f00 of size 512 next 103
2023-11-23 19:28:54.650242: I tensorflow/tsl/framework/bfc_allocator.cc:1095] InUse at 149724e04100 of size 512 next 102
2023-11-23 19:28:54.650246: I tensorflow/tsl/framework/bfc_allocator.cc:1095] InUse at 149724e04300 of size 512 next 101
2023-11-23 19:28:54.650251: I tensorflow/tsl/framework/bfc_allocator.cc:1095] InUse at 149724e04500 of size 512 next 100
2023-11-23 19:28:54.650256: I tensorflow/tsl/framework/bfc_allocator.cc:1095] InUse at 149724e04700 of size 512 next 97
2023-11-23 19:28:54.650261: I tensorflow/tsl/framework/bfc_allocator.cc:1095] InUse at 149724e04900 of size 512 next 96
2023-11-23 19:28:54.650265: I tensorflow/tsl/framework/bfc_allocator.cc:1095] InUse at 149724e04b00 of size 512 next 95
2023-11-23 19:28:54.650270: I tensorflow/tsl/framework/bfc_allocator.cc:1095] InUse at 149724e04d00 of size 512 next 90
2023-11-23 19:28:54.650274: I tensorflow/tsl/framework/bfc_allocator.cc:1095] InUse at 149724e04f00 of size 512 next 44
2023-11-23 19:28:54.650279: I tensorflow/tsl/framework/bfc_allocator.cc:1095] InUse at 149724e05100 of size 256 next 46
2023-11-23 19:28:54.650284: I tensorflow/tsl/framework/bfc_allocator.cc:1095] InUse at 149724e05200 of size 256 next 45
2023-11-23 19:28:54.650288: I tensorflow/tsl/framework/bfc_allocator.cc:1095] InUse at 149724e05300 of size 256 next 35
2023-11-23 19:28:54.650293: I tensorflow/tsl/framework/bfc_allocator.cc:1095] InUse at 149724e05400 of size 512 next 172
2023-11-23 19:28:54.650298: I tensorflow/tsl/framework/bfc_allocator.cc:1095] InUse at 149724e05600 of size 768 next 62
2023-11-23 19:28:54.650302: I tensorflow/tsl/framework/bfc_allocator.cc:1095] InUse at 149724e05900 of size 256 next 63
2023-11-23 19:28:54.650307: I tensorflow/tsl/framework/bfc_allocator.cc:1095] InUse at 149724e05a00 of size 256 next 64
2023-11-23 19:28:54.650312: I tensorflow/tsl/framework/bfc_allocator.cc:1095] InUse at 149724e05b00 of size 256 next 66
2023-11-23 19:28:54.650317: I tensorflow/tsl/framework/bfc_allocator.cc:1095] InUse at 149724e05c00 of size 256 next 67
2023-11-23 19:28:54.650321: I tensorflow/tsl/framework/bfc_allocator.cc:1095] InUse at 149724e05d00 of size 32768 next 60
2023-11-23 19:28:54.650326: I tensorflow/tsl/framework/bfc_allocator.cc:1095] InUse at 149724e0dd00 of size 12288 next 69
2023-11-23 19:28:54.650331: I tensorflow/tsl/framework/bfc_allocator.cc:1095] InUse at 149724e10d00 of size 12288 next 193
2023-11-23 19:28:54.650337: I tensorflow/tsl/framework/bfc_allocator.cc:1095] InUse at 149724e13d00 of size 32768 next 106
2023-11-23 19:28:54.650341: I tensorflow/tsl/framework/bfc_allocator.cc:1095] InUse at 149724e1bd00 of size 256 next 120
2023-11-23 19:28:54.650346: I tensorflow/tsl/framework/bfc_allocator.cc:1095] InUse at 149724e1be00 of size 1024 next 119
2023-11-23 19:28:54.650351: I tensorflow/tsl/framework/bfc_allocator.cc:1095] InUse at 149724e1c200 of size 1024 next 116
2023-11-23 19:28:54.650358: I tensorflow/tsl/framework/bfc_allocator.cc:1095] InUse at 149724e1c600 of size 1024 next 207
2023-11-23 19:28:54.650363: I tensorflow/tsl/framework/bfc_allocator.cc:1095] InUse at 149724e1ca00 of size 1024 next 113
2023-11-23 19:28:54.650368: I tensorflow/tsl/framework/bfc_allocator.cc:1095] InUse at 149724e1ce00 of size 1024 next 210
2023-11-23 19:28:54.650376: I tensorflow/tsl/framework/bfc_allocator.cc:1095] InUse at 149724e1d200 of size 1280 next 169
2023-11-23 19:28:54.650380: I tensorflow/tsl/framework/bfc_allocator.cc:1095] InUse at 149724e1d700 of size 256 next 52
2023-11-23 19:28:54.650385: I tensorflow/tsl/framework/bfc_allocator.cc:1095] InUse at 149724e1d800 of size 256 next 176
2023-11-23 19:28:54.650390: I tensorflow/tsl/framework/bfc_allocator.cc:1095] InUse at 149724e1d900 of size 256 next 202
2023-11-23 19:28:54.650395: I tensorflow/tsl/framework/bfc_allocator.cc:1095] InUse at 149724e1da00 of size 256 next 213
2023-11-23 19:28:54.650400: I tensorflow/tsl/framework/bfc_allocator.cc:1095] InUse at 149724e1db00 of size 7168 next 218
2023-11-23 19:28:54.650405: I tensorflow/tsl/framework/bfc_allocator.cc:1095] InUse at 149724e1f700 of size 512 next 88
2023-11-23 19:28:54.650410: I tensorflow/tsl/framework/bfc_allocator.cc:1095] InUse at 149724e1f900 of size 256 next 93
2023-11-23 19:28:54.650414: I tensorflow/tsl/framework/bfc_allocator.cc:1095] InUse at 149724e1fa00 of size 256 next 92
2023-11-23 19:28:54.650419: I tensorflow/tsl/framework/bfc_allocator.cc:1095] InUse at 149724e1fb00 of size 256 next 94
2023-11-23 19:28:54.650424: I tensorflow/tsl/framework/bfc_allocator.cc:1095] InUse at 149724e1fc00 of size 256 next 91
2023-11-23 19:28:54.650428: I tensorflow/tsl/framework/bfc_allocator.cc:1095] InUse at 149724e1fd00 of size 256 next 89
2023-11-23 19:28:54.650433: I tensorflow/tsl/framework/bfc_allocator.cc:1095] InUse at 149724e1fe00 of size 256 next 87
2023-11-23 19:28:54.650438: I tensorflow/tsl/framework/bfc_allocator.cc:1095] InUse at 149724e1ff00 of size 256 next 199
2023-11-23 19:28:54.650442: I tensorflow/tsl/framework/bfc_allocator.cc:1095] InUse at 149724e20000 of size 256 next 192
2023-11-23 19:28:54.650447: I tensorflow/tsl/framework/bfc_allocator.cc:1095] InUse at 149724e20100 of size 256 next 186
2023-11-23 19:28:54.650452: I tensorflow/tsl/framework/bfc_allocator.cc:1095] InUse at 149724e20200 of size 256 next 198
2023-11-23 19:28:54.650457: I tensorflow/tsl/framework/bfc_allocator.cc:1095] InUse at 149724e20300 of size 256 next 200
2023-11-23 19:28:54.650461: I tensorflow/tsl/framework/bfc_allocator.cc:1095] InUse at 149724e20400 of size 256 next 182
2023-11-23 19:28:54.650466: I tensorflow/tsl/framework/bfc_allocator.cc:1095] InUse at 149724e20500 of size 256 next 168
2023-11-23 19:28:54.650471: I tensorflow/tsl/framework/bfc_allocator.cc:1095] InUse at 149724e20600 of size 256 next 195
2023-11-23 19:28:54.650476: I tensorflow/tsl/framework/bfc_allocator.cc:1095] InUse at 149724e20700 of size 256 next 194
2023-11-23 19:28:54.650480: I tensorflow/tsl/framework/bfc_allocator.cc:1095] InUse at 149724e20800 of size 256 next 151
2023-11-23 19:28:54.650485: I tensorflow/tsl/framework/bfc_allocator.cc:1095] InUse at 149724e20900 of size 256 next 167
2023-11-23 19:28:54.650489: I tensorflow/tsl/framework/bfc_allocator.cc:1095] InUse at 149724e20a00 of size 256 next 140
2023-11-23 19:28:54.650494: I tensorflow/tsl/framework/bfc_allocator.cc:1095] InUse at 149724e20b00 of size 256 next 155
2023-11-23 19:28:54.650499: I tensorflow/tsl/framework/bfc_allocator.cc:1095] InUse at 149724e20c00 of size 256 next 142
2023-11-23 19:28:54.650503: I tensorflow/tsl/framework/bfc_allocator.cc:1095] InUse at 149724e20d00 of size 256 next 161
2023-11-23 19:28:54.650508: I tensorflow/tsl/framework/bfc_allocator.cc:1095] InUse at 149724e20e00 of size 256 next 216
2023-11-23 19:28:54.650513: I tensorflow/tsl/framework/bfc_allocator.cc:1095] InUse at 149724e20f00 of size 256 next 226
2023-11-23 19:28:54.650517: I tensorflow/tsl/framework/bfc_allocator.cc:1095] InUse at 149724e21000 of size 256 next 212
2023-11-23 19:28:54.650524: I tensorflow/tsl/framework/bfc_allocator.cc:1095] InUse at 149724e21100 of size 256 next 146
2023-11-23 19:28:54.650529: I tensorflow/tsl/framework/bfc_allocator.cc:1095] InUse at 149724e21200 of size 256 next 54
2023-11-23 19:28:54.650534: I tensorflow/tsl/framework/bfc_allocator.cc:1095] InUse at 149724e21300 of size 7168 next 99
2023-11-23 19:28:54.650539: I tensorflow/tsl/framework/bfc_allocator.cc:1095] InUse at 149724e22f00 of size 32768 next 150
2023-11-23 19:28:54.650543: I tensorflow/tsl/framework/bfc_allocator.cc:1095] InUse at 149724e2af00 of size 12288 next 181
2023-11-23 19:28:54.650548: I tensorflow/tsl/framework/bfc_allocator.cc:1095] InUse at 149724e2df00 of size 21504 next 109
2023-11-23 19:28:54.650553: I tensorflow/tsl/framework/bfc_allocator.cc:1095] InUse at 149724e33300 of size 256 next 174
2023-11-23 19:28:54.650558: I tensorflow/tsl/framework/bfc_allocator.cc:1095] InUse at 149724e33400 of size 256 next 9
2023-11-23 19:28:54.650563: I tensorflow/tsl/framework/bfc_allocator.cc:1095] InUse at 149724e33500 of size 1024 next 118
2023-11-23 19:28:54.650568: I tensorflow/tsl/framework/bfc_allocator.cc:1095] InUse at 149724e33900 of size 256 next 127
2023-11-23 19:28:54.650573: I tensorflow/tsl/framework/bfc_allocator.cc:1095] InUse at 149724e33a00 of size 256 next 126
2023-11-23 19:28:54.650577: I tensorflow/tsl/framework/bfc_allocator.cc:1095] InUse at 149724e33b00 of size 256 next 125
2023-11-23 19:28:54.650582: I tensorflow/tsl/framework/bfc_allocator.cc:1095] InUse at 149724e33c00 of size 1024 next 121
2023-11-23 19:28:54.650587: I tensorflow/tsl/framework/bfc_allocator.cc:1095] InUse at 149724e34000 of size 1024 next 128
2023-11-23 19:28:54.650592: I tensorflow/tsl/framework/bfc_allocator.cc:1095] InUse at 149724e34400 of size 1792 next 149
2023-11-23 19:28:54.650596: I tensorflow/tsl/framework/bfc_allocator.cc:1095] InUse at 149724e34b00 of size 256 next 55
2023-11-23 19:28:54.650601: I tensorflow/tsl/framework/bfc_allocator.cc:1095] InUse at 149724e34c00 of size 256 next 138
2023-11-23 19:28:54.650606: I tensorflow/tsl/framework/bfc_allocator.cc:1095] InUse at 149724e34d00 of size 256 next 158
2023-11-23 19:28:54.650611: I tensorflow/tsl/framework/bfc_allocator.cc:1095] InUse at 149724e34e00 of size 256 next 159
2023-11-23 19:28:54.650615: I tensorflow/tsl/framework/bfc_allocator.cc:1095] InUse at 149724e34f00 of size 256 next 147
2023-11-23 19:28:54.650620: I tensorflow/tsl/framework/bfc_allocator.cc:1095] InUse at 149724e35000 of size 256 next 152
2023-11-23 19:28:54.650625: I tensorflow/tsl/framework/bfc_allocator.cc:1095] InUse at 149724e35100 of size 256 next 156
2023-11-23 19:28:54.650630: I tensorflow/tsl/framework/bfc_allocator.cc:1095] InUse at 149724e35200 of size 256 next 132
2023-11-23 19:28:54.650634: I tensorflow/tsl/framework/bfc_allocator.cc:1095] InUse at 149724e35300 of size 256 next 8
2023-11-23 19:28:54.650639: I tensorflow/tsl/framework/bfc_allocator.cc:1095] InUse at 149724e35400 of size 256 next 131
2023-11-23 19:28:54.650644: I tensorflow/tsl/framework/bfc_allocator.cc:1095] InUse at 149724e35500 of size 256 next 130
2023-11-23 19:28:54.650648: I tensorflow/tsl/framework/bfc_allocator.cc:1095] InUse at 149724e35600 of size 256 next 80
2023-11-23 19:28:54.650653: I tensorflow/tsl/framework/bfc_allocator.cc:1095] InUse at 149724e35700 of size 256 next 117
2023-11-23 19:28:54.650658: I tensorflow/tsl/framework/bfc_allocator.cc:1095] InUse at 149724e35800 of size 256 next 115
2023-11-23 19:28:54.650663: I tensorflow/tsl/framework/bfc_allocator.cc:1095] InUse at 149724e35900 of size 1024 next 112
2023-11-23 19:28:54.650667: I tensorflow/tsl/framework/bfc_allocator.cc:1095] InUse at 149724e35d00 of size 1024 next 111
2023-11-23 19:28:54.650672: I tensorflow/tsl/framework/bfc_allocator.cc:1095] InUse at 149724e36100 of size 1024 next 110
2023-11-23 19:28:54.650677: I tensorflow/tsl/framework/bfc_allocator.cc:1095] InUse at 149724e36500 of size 1024 next 160
2023-11-23 19:28:54.650682: I tensorflow/tsl/framework/bfc_allocator.cc:1095] InUse at 149724e36900 of size 1024 next 154
2023-11-23 19:28:54.650688: I tensorflow/tsl/framework/bfc_allocator.cc:1095] InUse at 149724e36d00 of size 1024 next 178
2023-11-23 19:28:54.650693: I tensorflow/tsl/framework/bfc_allocator.cc:1095] InUse at 149724e37100 of size 1024 next 85
2023-11-23 19:28:54.650698: I tensorflow/tsl/framework/bfc_allocator.cc:1095] InUse at 149724e37500 of size 1024 next 86
2023-11-23 19:28:54.650703: I tensorflow/tsl/framework/bfc_allocator.cc:1095] InUse at 149724e37900 of size 1024 next 84
2023-11-23 19:28:54.650708: I tensorflow/tsl/framework/bfc_allocator.cc:1095] InUse at 149724e37d00 of size 1024 next 83
2023-11-23 19:28:54.650712: I tensorflow/tsl/framework/bfc_allocator.cc:1095] InUse at 149724e38100 of size 1024 next 82
2023-11-23 19:28:54.650717: I tensorflow/tsl/framework/bfc_allocator.cc:1095] InUse at 149724e38500 of size 1280 next 124
2023-11-23 19:28:54.650722: I tensorflow/tsl/framework/bfc_allocator.cc:1095] InUse at 149724e38a00 of size 18432 next 177
2023-11-23 19:28:54.650727: I tensorflow/tsl/framework/bfc_allocator.cc:1095] InUse at 149724e3d200 of size 256 next 53
2023-11-23 19:28:54.650731: I tensorflow/tsl/framework/bfc_allocator.cc:1095] InUse at 149724e3d300 of size 256 next 136
2023-11-23 19:28:54.650736: I tensorflow/tsl/framework/bfc_allocator.cc:1095] InUse at 149724e3d400 of size 256 next 141
2023-11-23 19:28:54.650741: I tensorflow/tsl/framework/bfc_allocator.cc:1095] InUse at 149724e3d500 of size 256 next 157
2023-11-23 19:28:54.650746: I tensorflow/tsl/framework/bfc_allocator.cc:1095] InUse at 149724e3d600 of size 256 next 171
2023-11-23 19:28:54.650750: I tensorflow/tsl/framework/bfc_allocator.cc:1095] InUse at 149724e3d700 of size 256 next 165
2023-11-23 19:28:54.650755: I tensorflow/tsl/framework/bfc_allocator.cc:1095] InUse at 149724e3d800 of size 14336 next 179
2023-11-23 19:28:54.650760: I tensorflow/tsl/framework/bfc_allocator.cc:1095] InUse at 149724e41000 of size 512 next 37
2023-11-23 19:28:54.650765: I tensorflow/tsl/framework/bfc_allocator.cc:1095] InUse at 149724e41200 of size 512 next 28
2023-11-23 19:28:54.650769: I tensorflow/tsl/framework/bfc_allocator.cc:1095] InUse at 149724e41400 of size 256 next 57
2023-11-23 19:28:54.650774: I tensorflow/tsl/framework/bfc_allocator.cc:1095] InUse at 149724e41500 of size 256 next 59
2023-11-23 19:28:54.650779: I tensorflow/tsl/framework/bfc_allocator.cc:1095] InUse at 149724e41600 of size 256 next 56
2023-11-23 19:28:54.650783: I tensorflow/tsl/framework/bfc_allocator.cc:1095] InUse at 149724e41700 of size 256 next 42
2023-11-23 19:28:54.650788: I tensorflow/tsl/framework/bfc_allocator.cc:1095] InUse at 149724e41800 of size 256 next 47
2023-11-23 19:28:54.650792: I tensorflow/tsl/framework/bfc_allocator.cc:1095] InUse at 149724e41900 of size 256 next 51
2023-11-23 19:28:54.650797: I tensorflow/tsl/framework/bfc_allocator.cc:1095] InUse at 149724e41a00 of size 512 next 16
2023-11-23 19:28:54.650802: I tensorflow/tsl/framework/bfc_allocator.cc:1095] InUse at 149724e41c00 of size 768 next 201
2023-11-23 19:28:54.650807: I tensorflow/tsl/framework/bfc_allocator.cc:1095] InUse at 149724e41f00 of size 256 next 163
2023-11-23 19:28:54.650811: I tensorflow/tsl/framework/bfc_allocator.cc:1095] InUse at 149724e42000 of size 256 next 135
2023-11-23 19:28:54.650816: I tensorflow/tsl/framework/bfc_allocator.cc:1095] InUse at 149724e42100 of size 256 next 144
2023-11-23 19:28:54.650821: I tensorflow/tsl/framework/bfc_allocator.cc:1095] InUse at 149724e42200 of size 256 next 229
2023-11-23 19:28:54.650825: I tensorflow/tsl/framework/bfc_allocator.cc:1095] InUse at 149724e42300 of size 256 next 184
2023-11-23 19:28:54.650830: I tensorflow/tsl/framework/bfc_allocator.cc:1095] InUse at 149724e42400 of size 256 next 220
2023-11-23 19:28:54.650835: I tensorflow/tsl/framework/bfc_allocator.cc:1095] InUse at 149724e42500 of size 262144 next 114
2023-11-23 19:28:54.650840: I tensorflow/tsl/framework/bfc_allocator.cc:1095] InUse at 149724e82500 of size 262144 next 134
2023-11-23 19:28:54.650845: I tensorflow/tsl/framework/bfc_allocator.cc:1095] InUse at 149724ec2500 of size 65536 next 81
2023-11-23 19:28:54.650852: I tensorflow/tsl/framework/bfc_allocator.cc:1095] InUse at 149724ed2500 of size 256 next 79
2023-11-23 19:28:54.650857: I tensorflow/tsl/framework/bfc_allocator.cc:1095] InUse at 149724ed2600 of size 256 next 78
2023-11-23 19:28:54.650862: I tensorflow/tsl/framework/bfc_allocator.cc:1095] InUse at 149724ed2700 of size 256 next 77
2023-11-23 19:28:54.650866: I tensorflow/tsl/framework/bfc_allocator.cc:1095] InUse at 149724ed2800 of size 256 next 76
2023-11-23 19:28:54.650871: I tensorflow/tsl/framework/bfc_allocator.cc:1095] InUse at 149724ed2900 of size 256 next 75
2023-11-23 19:28:54.650876: I tensorflow/tsl/framework/bfc_allocator.cc:1095] InUse at 149724ed2a00 of size 256 next 74
2023-11-23 19:28:54.650880: I tensorflow/tsl/framework/bfc_allocator.cc:1095] InUse at 149724ed2b00 of size 256 next 73
2023-11-23 19:28:54.650885: I tensorflow/tsl/framework/bfc_allocator.cc:1095] InUse at 149724ed2c00 of size 256 next 72
2023-11-23 19:28:54.650889: I tensorflow/tsl/framework/bfc_allocator.cc:1095] InUse at 149724ed2d00 of size 131072 next 29
2023-11-23 19:28:54.650895: I tensorflow/tsl/framework/bfc_allocator.cc:1095] InUse at 149724ef2d00 of size 131072 next 133
2023-11-23 19:28:54.650900: I tensorflow/tsl/framework/bfc_allocator.cc:1095] InUse at 149724f12d00 of size 220160 next 122
2023-11-23 19:28:54.650904: I tensorflow/tsl/framework/bfc_allocator.cc:1095] InUse at 149724f48900 of size 256 next 137
2023-11-23 19:28:54.650909: I tensorflow/tsl/framework/bfc_allocator.cc:1095] InUse at 149724f48a00 of size 256 next 153
2023-11-23 19:28:54.650914: I tensorflow/tsl/framework/bfc_allocator.cc:1095] InUse at 149724f48b00 of size 256 next 175
2023-11-23 19:28:54.650919: I tensorflow/tsl/framework/bfc_allocator.cc:1095] InUse at 149724f48c00 of size 256 next 183
2023-11-23 19:28:54.650923: I tensorflow/tsl/framework/bfc_allocator.cc:1095] InUse at 149724f48d00 of size 256 next 187
2023-11-23 19:28:54.650928: I tensorflow/tsl/framework/bfc_allocator.cc:1095] InUse at 149724f48e00 of size 256 next 190
2023-11-23 19:28:54.650933: I tensorflow/tsl/framework/bfc_allocator.cc:1095] InUse at 149724f48f00 of size 256 next 188
2023-11-23 19:28:54.650938: I tensorflow/tsl/framework/bfc_allocator.cc:1095] InUse at 149724f49000 of size 256 next 173
2023-11-23 19:28:54.650942: I tensorflow/tsl/framework/bfc_allocator.cc:1095] InUse at 149724f49100 of size 256 next 189
2023-11-23 19:28:54.650947: I tensorflow/tsl/framework/bfc_allocator.cc:1095] InUse at 149724f49200 of size 256 next 191
2023-11-23 19:28:54.650952: I tensorflow/tsl/framework/bfc_allocator.cc:1095] InUse at 149724f49300 of size 205312 next 71
2023-11-23 19:28:54.650957: I tensorflow/tsl/framework/bfc_allocator.cc:1095] InUse at 149724f7b500 of size 14336 next 129
2023-11-23 19:28:54.650961: I tensorflow/tsl/framework/bfc_allocator.cc:1095] InUse at 149724f7ed00 of size 116736 next 123
2023-11-23 19:28:54.650966: I tensorflow/tsl/framework/bfc_allocator.cc:1095] InUse at 149724f9b500 of size 65536 next 222
2023-11-23 19:28:54.650971: I tensorflow/tsl/framework/bfc_allocator.cc:1095] InUse at 149724fab500 of size 346880 next 18446744073709551615
2023-11-23 19:28:54.650984: I tensorflow/tsl/framework/bfc_allocator.cc:1075] Next region of size 16777216
2023-11-23 19:28:54.650989: I tensorflow/tsl/framework/bfc_allocator.cc:1095] InUse at 149725000000 of size 3870720 next 166
2023-11-23 19:28:54.650994: I tensorflow/tsl/framework/bfc_allocator.cc:1095] InUse at 1497253b1000 of size 196608 next 43
2023-11-23 19:28:54.650998: I tensorflow/tsl/framework/bfc_allocator.cc:1095] InUse at 1497253e1000 of size 196608 next 98
2023-11-23 19:28:54.651003: I tensorflow/tsl/framework/bfc_allocator.cc:1095] InUse at 149725411000 of size 256 next 240
2023-11-23 19:28:54.651008: I tensorflow/tsl/framework/bfc_allocator.cc:1095] InUse at 149725411100 of size 256 next 228
2023-11-23 19:28:54.651013: I tensorflow/tsl/framework/bfc_allocator.cc:1095] InUse at 149725411200 of size 256 next 107
2023-11-23 19:28:54.651020: I tensorflow/tsl/framework/bfc_allocator.cc:1095] InUse at 149725411300 of size 256 next 224
2023-11-23 19:28:54.651025: I tensorflow/tsl/framework/bfc_allocator.cc:1095] InUse at 149725411400 of size 256 next 148
2023-11-23 19:28:54.651030: I tensorflow/tsl/framework/bfc_allocator.cc:1095] InUse at 149725411500 of size 256 next 211
2023-11-23 19:28:54.651034: I tensorflow/tsl/framework/bfc_allocator.cc:1095] InUse at 149725411600 of size 256 next 197
2023-11-23 19:28:54.651039: I tensorflow/tsl/framework/bfc_allocator.cc:1095] InUse at 149725411700 of size 256 next 108
2023-11-23 19:28:54.651043: I tensorflow/tsl/framework/bfc_allocator.cc:1095] InUse at 149725411800 of size 256 next 242
2023-11-23 19:28:54.651048: I tensorflow/tsl/framework/bfc_allocator.cc:1095] InUse at 149725411900 of size 256 next 185
2023-11-23 19:28:54.651053: I tensorflow/tsl/framework/bfc_allocator.cc:1095] InUse at 149725411a00 of size 256 next 196
2023-11-23 19:28:54.651057: I tensorflow/tsl/framework/bfc_allocator.cc:1095] InUse at 149725411b00 of size 256 next 238
2023-11-23 19:28:54.651062: I tensorflow/tsl/framework/bfc_allocator.cc:1095] InUse at 149725411c00 of size 256 next 227
2023-11-23 19:28:54.651067: I tensorflow/tsl/framework/bfc_allocator.cc:1095] InUse at 149725411d00 of size 256 next 223
2023-11-23 19:28:54.651072: I tensorflow/tsl/framework/bfc_allocator.cc:1095] Free at 149725411e00 of size 1289728 next 209
2023-11-23 19:28:54.651076: I tensorflow/tsl/framework/bfc_allocator.cc:1095] InUse at 14972554cc00 of size 256 next 162
2023-11-23 19:28:54.651081: I tensorflow/tsl/framework/bfc_allocator.cc:1095] InUse at 14972554cd00 of size 1220608 next 217
2023-11-23 19:28:54.651086: I tensorflow/tsl/framework/bfc_allocator.cc:1095] InUse at 149725676d00 of size 24576 next 170
2023-11-23 19:28:54.651091: I tensorflow/tsl/framework/bfc_allocator.cc:1095] InUse at 14972567cd00 of size 4792320 next 180
2023-11-23 19:28:54.651096: I tensorflow/tsl/framework/bfc_allocator.cc:1095] InUse at 149725b0ed00 of size 10240 next 208
2023-11-23 19:28:54.651101: I tensorflow/tsl/framework/bfc_allocator.cc:1095] InUse at 149725b11500 of size 294912 next 204
2023-11-23 19:28:54.651106: I tensorflow/tsl/framework/bfc_allocator.cc:1095] InUse at 149725b59500 of size 1290240 next 164
2023-11-23 19:28:54.651112: I tensorflow/tsl/framework/bfc_allocator.cc:1095] Free at 149725c94500 of size 3586816 next 18446744073709551615
2023-11-23 19:28:54.651116: I tensorflow/tsl/framework/bfc_allocator.cc:1100] Summary of in-use Chunks by size:
2023-11-23 19:28:54.651123: I tensorflow/tsl/framework/bfc_allocator.cc:1103] 146 Chunks of size 256 totalling 36.5KiB
2023-11-23 19:28:54.651130: I tensorflow/tsl/framework/bfc_allocator.cc:1103] 19 Chunks of size 512 totalling 9.5KiB
2023-11-23 19:28:54.651135: I tensorflow/tsl/framework/bfc_allocator.cc:1103] 4 Chunks of size 768 totalling 3.0KiB
2023-11-23 19:28:54.651140: I tensorflow/tsl/framework/bfc_allocator.cc:1103] 20 Chunks of size 1024 totalling 20.0KiB
2023-11-23 19:28:54.651146: I tensorflow/tsl/framework/bfc_allocator.cc:1103] 3 Chunks of size 1280 totalling 3.8KiB
2023-11-23 19:28:54.651151: I tensorflow/tsl/framework/bfc_allocator.cc:1103] 1 Chunks of size 1792 totalling 1.8KiB
2023-11-23 19:28:54.651157: I tensorflow/tsl/framework/bfc_allocator.cc:1103] 2 Chunks of size 7168 totalling 14.0KiB
2023-11-23 19:28:54.651162: I tensorflow/tsl/framework/bfc_allocator.cc:1103] 1 Chunks of size 10240 totalling 10.0KiB
2023-11-23 19:28:54.651168: I tensorflow/tsl/framework/bfc_allocator.cc:1103] 3 Chunks of size 12288 totalling 36.0KiB
2023-11-23 19:28:54.651174: I tensorflow/tsl/framework/bfc_allocator.cc:1103] 2 Chunks of size 14336 totalling 28.0KiB
2023-11-23 19:28:54.651179: I tensorflow/tsl/framework/bfc_allocator.cc:1103] 1 Chunks of size 18432 totalling 18.0KiB
2023-11-23 19:28:54.651185: I tensorflow/tsl/framework/bfc_allocator.cc:1103] 1 Chunks of size 21504 totalling 21.0KiB
2023-11-23 19:28:54.651190: I tensorflow/tsl/framework/bfc_allocator.cc:1103] 1 Chunks of size 24576 totalling 24.0KiB
2023-11-23 19:28:54.651198: I tensorflow/tsl/framework/bfc_allocator.cc:1103] 3 Chunks of size 32768 totalling 96.0KiB
2023-11-23 19:28:54.651204: I tensorflow/tsl/framework/bfc_allocator.cc:1103] 2 Chunks of size 65536 totalling 128.0KiB
2023-11-23 19:28:54.651210: I tensorflow/tsl/framework/bfc_allocator.cc:1103] 1 Chunks of size 116736 totalling 114.0KiB
2023-11-23 19:28:54.651215: I tensorflow/tsl/framework/bfc_allocator.cc:1103] 2 Chunks of size 131072 totalling 256.0KiB
2023-11-23 19:28:54.651221: I tensorflow/tsl/framework/bfc_allocator.cc:1103] 2 Chunks of size 196608 totalling 384.0KiB
2023-11-23 19:28:54.651227: I tensorflow/tsl/framework/bfc_allocator.cc:1103] 1 Chunks of size 205312 totalling 200.5KiB
2023-11-23 19:28:54.651232: I tensorflow/tsl/framework/bfc_allocator.cc:1103] 1 Chunks of size 220160 totalling 215.0KiB
2023-11-23 19:28:54.651238: I tensorflow/tsl/framework/bfc_allocator.cc:1103] 2 Chunks of size 262144 totalling 512.0KiB
2023-11-23 19:28:54.651244: I tensorflow/tsl/framework/bfc_allocator.cc:1103] 1 Chunks of size 294912 totalling 288.0KiB
2023-11-23 19:28:54.651250: I tensorflow/tsl/framework/bfc_allocator.cc:1103] 1 Chunks of size 346880 totalling 338.8KiB
2023-11-23 19:28:54.651255: I tensorflow/tsl/framework/bfc_allocator.cc:1103] 1 Chunks of size 1220608 totalling 1.16MiB
2023-11-23 19:28:54.651261: I tensorflow/tsl/framework/bfc_allocator.cc:1103] 1 Chunks of size 1290240 totalling 1.23MiB
2023-11-23 19:28:54.651266: I tensorflow/tsl/framework/bfc_allocator.cc:1103] 1 Chunks of size 3870720 totalling 3.69MiB
2023-11-23 19:28:54.651272: I tensorflow/tsl/framework/bfc_allocator.cc:1103] 1 Chunks of size 4792320 totalling 4.57MiB
2023-11-23 19:28:54.651277: I tensorflow/tsl/framework/bfc_allocator.cc:1103] 1 Chunks of size 11780096 totalling 11.23MiB
2023-11-23 19:28:54.651283: I tensorflow/tsl/framework/bfc_allocator.cc:1103] 2 Chunks of size 11796480 totalling 22.50MiB
2023-11-23 19:28:54.651289: I tensorflow/tsl/framework/bfc_allocator.cc:1103] 1 Chunks of size 16777216 totalling 16.00MiB
2023-11-23 19:28:54.651294: I tensorflow/tsl/framework/bfc_allocator.cc:1103] 1 Chunks of size 21757952 totalling 20.75MiB
2023-11-23 19:28:54.651300: I tensorflow/tsl/framework/bfc_allocator.cc:1103] 1 Chunks of size 31752192 totalling 30.28MiB
2023-11-23 19:28:54.651306: I tensorflow/tsl/framework/bfc_allocator.cc:1107] Sum Total of in-use chunks: 114.11MiB
2023-11-23 19:28:54.651312: I tensorflow/tsl/framework/bfc_allocator.cc:1109] Total bytes in pool: 136314880 memory_limit_: 15312289792 available bytes: 15175974912 curr_region_allocation_bytes_: 8589934592
2023-11-23 19:28:54.651320: I tensorflow/tsl/framework/bfc_allocator.cc:1114] Stats:
Limit: 15312289792
InUse: 119658240
MaxInUse: 2897541120
NumAllocs: 26943
MaxAllocSize: 2743205888
Reserved: 0
PeakReserved: 0
LargestFreeBlock: 0
2023-11-23 19:28:54.651338: W tensorflow/tsl/framework/bfc_allocator.cc:497] *********________***************************xxxxx******************xxxxxx*********xxxx*********x**__
2023-11-23 19:28:54.651355: W tensorflow/core/framework/op_kernel.cc:1828] OP_REQUIRES failed at conv_ops_impl.h:588 : RESOURCE_EXHAUSTED: OOM when allocating tensor with shape[32,64,1,1439] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc
Traceback (most recent call last):
File "/home/rqchia/.local/lib/python3.8/site-packages/keras_tuner/src/engine/base_tuner.py", line 270, in _try_run_and_update_trial
self._run_and_update_trial(trial, *fit_args, **fit_kwargs)
File "/home/rqchia/.local/lib/python3.8/site-packages/keras_tuner/src/engine/base_tuner.py", line 235, in _run_and_update_trial
results = self.run_trial(trial, *fit_args, **fit_kwargs)
File "/home/rqchia/.local/lib/python3.8/site-packages/keras_tuner/src/engine/tuner.py", line 314, in run_trial
obj_value = self._build_and_fit_model(trial, *args, **copied_kwargs)
File "/home/rqchia/.local/lib/python3.8/site-packages/keras_tuner/src/engine/tuner.py", line 233, in _build_and_fit_model
results = self.hypermodel.fit(hp, model, *args, **kwargs)
File "/home/rqchia/projects/aria-respiration-cal/models/neuralnet.py", line 178, in fit
history = model.fit(x, y,