New research suggests AI image generation using DALL-E2 has promising future in radiology
Figure 2. Examples of text-to-image–generated anatomical structures in CT, MRI, and ultrasound images created with DALL-E 2. CT: computed tomography; MRI: magnetic resonance imaging.To investigate the extent of DALL-E 2’s radiological knowledge, we tested how well the modal can reconstruct missing areas in a radiological image .
To do this, we selected radiographs of the pelvis, ankle, chest, shoulder, knee, wrist, and thoracic spine and erased specific areas before handing the remnants to DALL-E 2 for reconstruction . The accompanying prompts were identical to the those used for generating the text-based images described in thesection. DALL-E 2 provided realistic replacement images that were nearly indistinguishable from the original radiographs of the pelvis, thorax, and thoracic spine. However, the results were not as convincing when a joint was included in the image area. In the ankle and wrist images, the number of tarsal bones and the structure varied greatly from those in the original radiographs and deviated from realistic representations. For the shoulder images, DALL-E 2 failed to reconstruct the glenoid cavity and articular surface of the humerus. In one image, a foreign body was inserted into the shoulder that remotely resembled a prosthesis. Further, when reconstructing the knee image, the model omitted the patella but retained the bicondylar structure of the femur. Figure 3. Reconstructed areas of different anatomical locations in x-rays created by using DALL-E 2. The yellow-bordered regions of the original images were erased before providing the remnant images for reconstruction.Since the previous two experiments showed that DALL-E 2 could handle the basics of standard radiological images, we also wanted to investigate how well anatomical knowledge is embedded in the model. To do this, we randomly selected radiographic images of the abdomen, chest, pelvis, knee, various spinal regions, wrist bones, and hand and had DALL-E 2 extend these images beyond their boundaries, since the model needs to know both locations and anatomical proportions for this task. The accompanying prompts were selected based on the anatomical regions to be created. Again, the style of the augmented images represented a realistic representation of radiographs . It was possible to create a complete full-body radiograph by using only 1 image of the knee as a starting point. The greater the distance between the original image and the generated area, the less detailed the images became. Anatomical proportions, such as the length of the femur or the size of the lung, remained realistic, but finer details, such as the number of lumbar vertebrae, were inconsistent. The model generally performed best when creating anterior and posterior views, while the creation of lateral views was more challenging and produced poorer results. Figure 4. Extending x-ray images of different anatomical regions beyond their borders by using DALL-E 2. The original x-rays are shown in yellow boxes, and the areas outside of the yellow boxes were generated by DALL-E 2.The generation of pathological images to, for example, visualize fractures, intracranial hemorrhages, or tumors was limited. We tested this by generating fractures on radiographs, which mostly resulted in distorted images that were similar to those in . In addition, DALL-E 2 has a filter that prevents the generation of harmful content. Therefore, we could not test most pathologies because words such as “bleeding” triggered this filter.We were able to show that DALL-E 2 can generate x-ray images that are similar to authentic x-ray images in terms of style and anatomical proportions. Thus, we conclude that relevant representations for x-rays were learned by DALL-E 2 during training. However, it seems that only representations for radiographs without pathologies are available or are allowed by the filter, as the generation of pathological images was limited. In addition, DALL-E 2’s generative capabilities were poor for CT, MRI, and ultrasound images. Access to data is critical in deep learning, and in general, the larger the data set, the better the performance of the models trained on the data set. However, especially in radiology, there is no single large database from which to create such a data set. Instead, the necessary data are divided among several institutions, and privacy concerns additionally prevent the merging of these data. Synthetic data from generative models, such as DALL-E 2, show promise for addressing these issues by enabling the creation of data sets that are much larger than those that are currently available and greatly accelerating the development of new deep learning tools for radiology [
United Kingdom Latest News, United Kingdom Headlines
Similar News:You can also read news stories similar to this one that we have collected from other news sources.
What do we know about the Silicon Valley and Signature Bank collapse?Two US banks have collapsed and shares tumbled. So how serious is this and what does it mean for you?
Read more »
'Our little girl went to sleep healthy, but never woke up - we don't know why''Our little girl went to sleep healthy, but she never woke up and we don't know why'
Read more »
Terrifying moment smiling murderer tells police 'I know what happened' as he returns to crime sceneA drunk killer returned to the crime scene where he murdered a 'brilliant' dad-of-two in Chelmsford, Essex, offering to help police.
Read more »
US drone crash: What do we know about collision with Russian jet?Defence analyst Professor Michael Clarke tells Sky News the Russian fighter jet's contact with the propeller of a US drone was likely an accident.
Read more »
Energy bills support has been extended - here's what you need to knowJeremy Hunt will cancel the £500 hike in average energy bills that was about to hit households, but critics have branded his failure to actually reduce bills as ‘truly pathetic’ ⬇️ kt_grant explains
Read more »
What to Know About ChatGPT-4 and How to Use It Right NowIt’s goodbye to ChatGPT and hello ChatGPT-4—an even more powerful tool, sure to send even bigger ripples across the world. So what is GPT-4, and how can you use it? Here’s everything you need to know
Read more »