# Unable to re-run the notebook

**URL:** <https://discourse.canceridc.dev/t/unable-to-re-run-the-notebook/194>\
**Category:** Support\
**Tags:** ai-notebooks\
**Created:** [September 2, 2021, 10:14pm UTC](https://discourse.canceridc.dev/t/unable-to-re-run-the-notebook/194 "2021-09-02T22:14:43Z")\
**Posts on this page:** 9\
**Page:** 1

<div class="post-metadata">

**Author:** ![giemmecci](https://sea1.discourse-cdn.com/flex015/user_avatar/discourse.canceridc.dev/giemmecci/32/71_2.png) [@giemmecci](https://discourse.canceridc.dev/u/giemmecci)\
**Post date:** [September 2, 2021, 10:14pm UTC](https://discourse.canceridc.dev/t/unable-to-re-run-the-notebook/194/1 "2021-09-02T22:14:43Z")

</div>

Hi, I was trying to re-run my notebook but I’m getting the following error message when I try to run a segmentation model using docker

```auto
THCudaCheck FAIL file=/pytorch/aten/src/THC/THCGeneral.cpp line=47 error=999 : unknown error
Traceback (most recent call last):
  File "/usr/local/bin/hd-bet", line 7, in <module>
    exec(compile(f.read(), __file__ , 'exec'))
  File "/HD-BET/HD_BET/hd-bet", line 119, in <module>
    run_hd_bet(input_files, output_files, mode, config_file, device, pp, tta, save_mask, overwrite_existing)
  File "/HD-BET/HD_BET/run.py", line 63, in run_hd_bet
    net.cuda(device)
  File "/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py", line 458, in cuda
    return self._apply(lambda t: t.cuda(device))
  File "/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py", line 354, in _apply
    module._apply(fn)
  File "/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py", line 376, in _apply
    param_applied = fn(param)
  File "/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py", line 458, in <lambda>
    return self._apply(lambda t: t.cuda(device))
  File "/usr/local/lib/python3.6/dist-packages/torch/cuda/ __init__.py", line 190, in _lazy_init
    torch._C._cuda_init()
RuntimeError: cuda runtime error (999) : unknown error at /pytorch/aten/src/THC/THCGeneral.cpp:47
Using contrast T1 as reference
Traceback (most recent call last):
  File "scripts/run.py", line 505, in <module>
    not args.no_permissions
  File "scripts/run.py", line 280, in run
    output1 = subp.check_output(["hd-bet", "-i", file_, "-device", "0"])
  File "/usr/lib/python3.6/subprocess.py", line 356, in check_output
    **kwargs).stdout
  File "/usr/lib/python3.6/subprocess.py", line 438, in run
    output=stdout, stderr=stderr)
subprocess.CalledProcessError: Command '['hd-bet', '-i', '/output/T1_r2s.nii.gz', '-device', '0']' returned non-zero exit status 1.

```

I’ve tried looking for solutions online, but nothing worked. I didn’t make any change to the environment or my virtual machine, and the same code works just fine on another virtual machine on the same Google Cloud project.

Any clue of what could be the issue?

Thanks!

---

<div class="post-metadata">

**Author:** ![aptekarev](https://avatars.discourse-cdn.com/v4/letter/a/dfb087/32.png) [@aptekarev](https://discourse.canceridc.dev/u/aptekarev)\
**Post date:** [September 3, 2021, 2:45pm UTC](https://discourse.canceridc.dev/t/unable-to-re-run-the-notebook/194/2 "2021-09-03T14:45:04Z")

</div>

With the log it looks like the problem is with the GPU. Either outdated drivers or a general configuration error. Is your VM a managed notebook instance?

---

<div class="post-metadata">

**Author:** ![fedorov](https://sea1.discourse-cdn.com/flex015/user_avatar/discourse.canceridc.dev/fedorov/32/3_2.png) [@fedorov](https://discourse.canceridc.dev/u/fedorov)\
**Post date:** [September 3, 2021, 5:00pm UTC](https://discourse.canceridc.dev/t/unable-to-re-run-the-notebook/194/3 "2021-09-03T17:00:56Z")

</div>

@giemmecci can you provide the details about what is going on:

- are you using AI Notebooks VM and not Colab?
- are you saying the exact same VM/notebook that worked in the past does not work, and you didn’t apply any system updates to the VM?
- can you confirm which notebook you are having troubles with? I tried to run this notebook [https://github.com/giemmecci/IDC/blob/main/GBM\_IDH1\_mutation\_radiomic\_classifier-Copy-GCP.ipynb](https://github.com/giemmecci/IDC/blob/main/GBM_IDH1_mutation_radiomic_classifier-Copy-GCP.ipynb), but I am not sure it is ready to use. I see that you check out [https://github.com/dicomsort/dicomsort](https://github.com/dicomsort/dicomsort), which is a completely different tool from [https://github.com/pieper/dicomsort](https://github.com/pieper/dicomsort), and so where you try to run `dicomsort/dicomsort.py` is not expected to work. I am not sure if I am looking at the right notebook.

---

<div class="post-metadata">

**Author:** ![giemmecci](https://sea1.discourse-cdn.com/flex015/user_avatar/discourse.canceridc.dev/giemmecci/32/71_2.png) [@giemmecci](https://discourse.canceridc.dev/u/giemmecci)\
**Post date:** [September 3, 2021, 10:19pm UTC](https://discourse.canceridc.dev/t/unable-to-re-run-the-notebook/194/4 "2021-09-03T22:19:36Z")

</div>

> [@aptekarev](#):
>
> With the log it looks like the problem is with the GPU. Either outdated drivers or a general configuration error. Is your VM a managed notebook instance?

Yes, it’s definitely something with the GPU; I’ve tried looking for the error online, and people seem to fix it either by [re-installing PyTorch](https://github.com/pytorch/pytorch/issues/15797) (which is not the solution for me since I haven’t installed it on my VM; the error arises when trying to apply a segmentation model using docker) or [reloading the NVIDIA kernel](https://discuss.pytorch.org/t/cuda-runtime-error-999/69658/12).  
I’m not sure if this will answer your question, but I’m launching the notebook from the AI Platform section (project: idc-external-005, instance gm-tf2).

> [@TCGA-GBM tutorial notebook](https://discourse.canceridc.dev/t/tcga-gbm-tutorial-notebook/162/30):
>
> - are you using AI Notebooks VM and not Colab?  
> Yes, AI Notebooks VM, no Colab.
> - are you saying the exact same VM/notebook that worked in the past does not work, and you didn’t apply any system updates to the VM?  
> Yes; I didn’t update the VM, and I was able to run the docker with the segmentation model in the past. I’ve tried running the notebook on another VM on the same project (kevin-ml) and it worked.
> - can you confirm which notebook you are having troubles with? I tried to run this notebook [https://github.com/giemmecci/IDC/blob/main/GBM\_IDH1\_mutation\_radiomic\_classifier-Copy-GCP.ipynb](https://github.com/giemmecci/IDC/blob/main/GBM_IDH1_mutation_radiomic_classifier-Copy-GCP.ipynb), but I am not sure it is ready to use. I see that you check out [GitHub - dicomsort/dicomsort: DICOM sorting utility](https://github.com/dicomsort/dicomsort), which is a completely different tool from [GitHub - pieper/dicomsort: A project to provide custom sorting and renaming of dicom files](https://github.com/pieper/dicomsort), and so where you try to run `dicomsort/dicomsort.py` is not expected to work. I am not sure if I am looking at the right notebook.  
> Oh, good catch! I didn’t catch the dicomsort error because I had already installed the correct version before starting working on this notebook.  
> I’ve created an updated version of the notebook that should run fine from scratch; you can find it [here](https://github.com/giemmecci/IDC/blob/main/TCGA-GBM_test.ipynb).

Thank you for your help and your time!

---

<div class="post-metadata">

**Author:** ![aptekarev](https://avatars.discourse-cdn.com/v4/letter/a/dfb087/32.png) [@aptekarev](https://discourse.canceridc.dev/u/aptekarev)\
**Post date:** [September 5, 2021, 9:53am UTC](https://discourse.canceridc.dev/t/unable-to-re-run-the-notebook/194/5 "2021-09-05T09:53:35Z")

</div>

> [@giemmecci](#):
>
> I’m launching the notebook from the AI Platform section

Since those are managed instances - you can just create a new one and check if it works in the new instance. If it does work my guess would be that the error you see on an older instance is related to the recent merger of the AI Notebooks into the newer Vertex AI product in Google Cloud.

---

<div class="post-metadata">

**Author:** ![giemmecci](https://sea1.discourse-cdn.com/flex015/user_avatar/discourse.canceridc.dev/giemmecci/32/71_2.png) [@giemmecci](https://discourse.canceridc.dev/u/giemmecci)\
**Post date:** [September 5, 2021, 9:37pm UTC](https://discourse.canceridc.dev/t/unable-to-re-run-the-notebook/194/6 "2021-09-05T21:37:57Z")

</div>

Thanks for the suggestion, but unfortunately it didn’t work; I’ve created another instance with the same characteristics as my older one, but I’m still getting the same error message:

```auto
09:30:23 PM: Reading series 1.3.6.1.4.1.14519.5.2.1.1706.4001.103174687731052142735983046836
09:30:23 PM: Reading series 1.3.6.1.4.1.14519.5.2.1.1706.4001.287753826073472752590065451465
09:30:23 PM: Reading series 1.3.6.1.4.1.14519.5.2.1.1706.4001.261091685370401963952512175444
09:30:23 PM: Reading series 1.3.6.1.4.1.14519.5.2.1.1706.4001.116289242878713280880004720679
THCudaCheck FAIL file=/pytorch/aten/src/THC/THCGeneral.cpp line=47 error=999 : unknown error
Traceback (most recent call last):
  File "/usr/local/bin/hd-bet", line 7, in <module>
    exec(compile(f.read(), __file__ , 'exec'))
  File "/HD-BET/HD_BET/hd-bet", line 119, in <module>
    run_hd_bet(input_files, output_files, mode, config_file, device, pp, tta, save_mask, overwrite_existing)
  File "/HD-BET/HD_BET/run.py", line 63, in run_hd_bet
    net.cuda(device)
  File "/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py", line 458, in cuda
    return self._apply(lambda t: t.cuda(device))
  File "/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py", line 354, in _apply
    module._apply(fn)
  File "/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py", line 376, in _apply
    param_applied = fn(param)
  File "/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py", line 458, in <lambda>
    return self._apply(lambda t: t.cuda(device))
  File "/usr/local/lib/python3.6/dist-packages/torch/cuda/ __init__.py", line 190, in _lazy_init
    torch._C._cuda_init()
RuntimeError: cuda runtime error (999) : unknown error at /pytorch/aten/src/THC/THCGeneral.cpp:47
Using contrast T1 as reference
Traceback (most recent call last):
  File "scripts/run.py", line 505, in <module>
    not args.no_permissions
  File "scripts/run.py", line 280, in run
    output1 = subp.check_output(["hd-bet", "-i", file_, "-device", "0"])
  File "/usr/lib/python3.6/subprocess.py", line 356, in check_output
    **kwargs).stdout
  File "/usr/lib/python3.6/subprocess.py", line 438, in run
    output=stdout, stderr=stderr)
subprocess.CalledProcessError: Command '['hd-bet', '-i', '/output/T1_r2s.nii.gz', '-device', '0']' returned non-zero exit status 1.

```

---

<div class="post-metadata">

**Author:** ![giemmecci](https://sea1.discourse-cdn.com/flex015/user_avatar/discourse.canceridc.dev/giemmecci/32/71_2.png) [@giemmecci](https://discourse.canceridc.dev/u/giemmecci)\
**Post date:** [September 5, 2021, 9:39pm UTC](https://discourse.canceridc.dev/t/unable-to-re-run-the-notebook/194/7 "2021-09-05T21:39:39Z")

</div>

> [@fedorov](#):
>
> Oh, good catch! I didn’t catch the dicomsort error because I had already installed the correct version before starting working on this notebook.  
> I’ve created an updated version of the notebook that should run fine from scratch; you can find it [here](https://github.com/giemmecci/IDC/blob/main/TCGA-GBM_test_short.ipynb).

I’ve updated the notebook to make the test shorter (reduced the number of MRIs that are downloaded)

---

<div class="post-metadata">

**Author:** ![denbonte](https://sea1.discourse-cdn.com/flex015/user_avatar/discourse.canceridc.dev/denbonte/32/133_2.png) [@denbonte](https://discourse.canceridc.dev/u/denbonte)\
**Post date:** [October 8, 2021, 9:32am UTC](https://discourse.canceridc.dev/t/unable-to-re-run-the-notebook/194/8 "2021-10-08T09:32:19Z")

</div>

Hey @giemmecci,

I just documented [in another thread](https://discourse.canceridc.dev/t/cuda-error-after-updating-ai-notebook-vm/223) what might be a solution for your problem. Your case looks different (as the error CUDA raises in your case is a `RuntimeError` and not an `initialization error`) - but it’s worth a shot (especially because re-installing NVIDIA drivers usually solves most of the CUDA issues anyways).

If you still have that instance paused, could you try the fix and let us know if that works? If that doesn’t solve it, feel free to come back with some logs. I would be happy to try and troubleshoot this (and curious to see if we can manage to replicate the issue).

Thanks,  
Dennis.

---

<div class="post-metadata">

**Author:** ![giemmecci](https://sea1.discourse-cdn.com/flex015/user_avatar/discourse.canceridc.dev/giemmecci/32/71_2.png) [@giemmecci](https://discourse.canceridc.dev/u/giemmecci)\
**Post date:** [October 11, 2021, 8:43pm UTC](https://discourse.canceridc.dev/t/unable-to-re-run-the-notebook/194/9 "2021-10-11T20:43:25Z")

</div>

Hi! Thanks so much for the hint! I’ll give it a try!