Hey everyone!
Sharing some observations.
Here is a comparison between the outputs of -
Our custom v-net model which has an accuracy (dice co-efficient) of ~95% for lung and tumor segmentations on the RADIOMICS dataset (vetted by a USBC radiologist)
AND
Meta’s SAM on the same CT scans from RADIOMICS
I used the “Everything” option in SAM and while it seems to do well on the lung segmentation, it almost never identified the tumor in any of the “test” scans that were fed to it. The other “classes” it marks don’t really represent anything (organ/nodule/node) in most cases.
Looks like SAM needs significant refining before being able to segment out lung cancer CT scans satisfactorily.
I also tried the “Box” method and unless the bounding box encompasses the whole tumor(s) precisely, SAM seems to incorrectly mark the tumor.
(The left part of the image is our model’s output - it has 2 classes and the right part is SAM’s output)
I want to upload more comparison images but I am restricted to just one.
2 Likes
@Aparna_Prabhu thank you for sharing these findings, this is very interesting! What was your experience with evaluating consistency of the segmentation across adjacent slices? Can you share some screenshots with the out of plane reformats?
I bumped your trust level in the forum - hopefully this will make it possible for you to share more than one image.
2 Likes
Surely Dr Fedorov. @fedorov
While I collate some results from out of plane reformats,
We, the ML team from iMerit Technology have a couple of observations to share -
Across adjacent slices in RADIOMICS, SAM continues to not identify the tumor - and this is regardless of the tumor size, location and whether or not it has metastasized.
Our model shows very high accuracy for lungs and identifies the tumor in 9 out of 10 slices in any scan from RADIOMICS.
I am attaching a couple of diverse results. Thank you so much for allowing me to post more than one image.
The perception, size and location of the tumor are widely different in either image.
The left portion is our model output while the right portion is from SAM.
Our lung class is represented by blue and the tumor class by pink.
From our experience, what helped attain good accuracy in tumor segmentation is high quality, diverse and voluminous ground truth (GT) that serves as training data for the model.
As you know, RADIOMICS contains annotations on some of its images. We got a board certified radiologist to evaluate these annotations and suggest changes as needed. The changes were mostly tiny and not obvious to people outside the medical fraternity but they made the biggest difference to model accuracy as we were watching the boundaries of the segmentations down to the last pixel.
After incorporating these changes manually (using 3D slicer in some cases and V7 in others), we were hitting 95% dice and <0.05 cross entropy loss in most cases.
For slices captured from the viewpoints of the kinds I have attached, SAM failed to even recognize the lungs. That was a bit off-putting because frankly, high accuracy in lung segmentations is not super hard to attain.
Looking forward to your thoughts.
Best,
Aparna Prabhu
ML lead, imerit.net
@Aparna_Prabhu it is interesting to see these findings, your progress sounds promising, and it is definitely a strength that you have a board-certified radiologist assisting you.
I guess my question is - is there anything you need from IDC to help you get the data, search the data, or anything else? Or you are sharing those findings just for the reference of the IDC community?
