Accuracy

Accuracy is an interesting thing in medicine. There is an assumption, this unspoken hope, from those who seek medical care that the answer they get is correct. Certainty breeds confidence, and confidence (without reference to accuracy) breeds a measure of certainty. A diagnosis, confidently stated can lead to a feeling of reassurance, and that the answer/the diagnosis is accurate. Physicians are intensely trained, and have a high level of professionally acquired skill. However, medical practitioners are not perfect. The idea that medical purveyors are inacurate, and do not come up with the correct answer the first time (or the second or third time) propels the success of television dramas like House, M.D.

When thinking about curating a medical database of many user submitted images, the topic of accuracy arises. Can we trust a citizen, a user, a non-professional to contribute an image of a skin lesion? Will it be labeled with the correct diagnosis? In my mind this is an open question. As described previously, we like to share images - think about the number of pictures uploaded to Facebook daily. The number of encounters with medical professionals related to skin issues far outstrips the number of pictures in currently available databases (see prior posts). It is the belief of this project that people can be trusted to contribute to a scientific project in a meaningful (meaning accurate) way. As a first pass, the uploaded image will rely on the diagnostic accuracy of the medical professional who evaluates the rash or lesion. It is trust given to the user/image contributor that they are contributing what a medical professional has communicated.

The question that has been in my mind recently – How accurate are medical professionals when pronouncing a diagnosis regarding the skin?

I conducted a review in the published scientific literature and come across some information shared below which will shed some light on this question. This is by no means an exhaustive review. If you are interested in the full paper please email me, however they are easily accessible via Pubmed. Below you will find a listing of medical/scientific studies from the 1970s through to the present which look into this question. The year of the study publication is listed as well as the title of the medical paper and a brief description along with an accuracy rate in the text. In many cases accuracy is defined as the diagnosis given to the picture or patient after examination as compared to the correct diagnosis which is typically a biopsy-which serves as the final arbiter of correctness in skin diagnosis.

1972    Accuracy of Dermatologic Diagnosis by Television

Comparisons of diagnoses made by black and white television viewing of slides of dermatologic lesions to those made from directly viewing of the same slides in color revealed that in 85% to 89% of cases the dermatologists were as accurate by television as on direct examination. Color television improved accuracy only slightly, but was more acceptable to the dermatologists as less time was required to reach a diagnosis.

1983    The Prevalence and Accuracy of Diagnosis of Non-Melanotic Skin Cancer in Victoria

Surveys of Victorian [Australian] dermatologists and pathologists were undertaken to determine the number of patients attending medical practitioners with non-melanotic [non-cancer] skin cancers and solar keratoses. Accuracy of clinical diagnosis studies suggest that the correct diagnosis of these tumors is being made clinically [in person] in approximately 70% of cases by experienced clinicians.

1989    The Development of Expertise in Dermatology

To examine the development of expertise in dermatology at five levels of expertise. A total of 100 slides [pictures], 2 typical and 3 atypical, from each of 20 common skin disorders, were presented to six subjects at each of the following levels: second-year preclinical medical students, final year medical students, residents in family medicine, general practitioners, and dermatologists. Accuracy of diagnosis rose from 21% for medical students to 87% for dermatologists.

1990    Accuracy in the Clinical Diagnosis of Malignant Melanoma

The computerized database (1955 through 1982) of the Oncology Section of the Skin and Cancer Unit of New York (NY) University Medical Center includes data on 13,878 lesions. Of these lesions, 214 were diagnosed clinically and histologically as malignant melanoma (MM)...The diagnostic accuracy for the best period (1974 - 1982) was 64%. The diagnosis of MM was made in 84.5% of the histologically proved cases of MM, reflecting a high degree of sensitivity.

 

2001    A comparison of dermatologists' and primary care physicians' accuracy in diagnosing melanoma: a systematic review

Studies were evaluated to determine the sensitivity and specificity of dermatologists' or PCPs' [Primary Care Providers - also known as general practitioners, commonly family doctors or internal medicine doctors in the US] ability to correctly diagnose lesions suggestive of melanoma and to perform biopsies or refer patients with such lesions. For diagnostic accuracy, sensitivity was 0.81 to 1.00 for dermatologists and 0.42 to 1.00 for PCPs. None of the studies reported specificity for dermatologists; one reported specificity for PCPs (0.98). For biopsy or referral accuracy, sensitivity ranged from 0.82 to 1.00 for dermatologists and 0.70 to 0.88 for PCPs; specificity, 0.70 to 0.89 for dermatologists and 0.70 to 0.87 for PCPs. Receiver operating characteristic curves for biopsy or referral ability were inconclusive. The study concluded that published data are inadequate to demonstrate differences in dermatologists' and PCPs' diagnostic and biopsy or referral accuracy of lesions suggestive of melanoma.

2003    Comparison of diagnostic accuracy for cutaneous malignant melanoma between general dermatology, plastic surgery and pigmented lesion clinics.

Since the 1980s there have been dedicated pigmented lesion clinics (PLCs) in the U.K. This study compared the false-negative rate (FNR) of clinical diagnosis with other clinics of primary referral of malignant melanoma (MM) in the same geographical area.

The case notes of 731 patients were available, of whom approximately two-thirds initially attended the PLC, one-fifth the General Dermatology clinics (D) and the remainder were divided approximately between Plastic Surgery clinics (P), other clinics (O) and the general practitioner (GP). The FNR was lowest for the PLC, at 10%, compared with 29% (D), 19% (P), 55% (O) and 54% (GP) (P < 0.0001).

**Accuraccy rates can be considered 100 - percentage above (ie: Accuracy at the dermatology clinic: 100 - 29% false negatives = 71% accurate/true positive diagnosis.

2003    Pattern analysis, not simplified algorithms, is the most reliable method for teaching dermoscopy for melanoma diagnosis to residents in dermatology

This study investigated the diagnostic performance of three different methods of teaching dermoscopy when used by newly trained residents in dermatology to diagnose melanocytic [cancerous] lesions. Pattern analysis yielded the best mean diagnostic accuracy (68.7%), followed by the ABCD rule (56.1%) and the seven-point check-list (53.4%, P = 0·06).

2004   A retrospective biopsy study of the clinical diagnostic accuracy of common skin diseases by different specialties compared with dermatology

The clinical diagnoses of family physicians, plastic, general, and orthopedic surgeons, and internists and pediatricians versus dermatologists were correlated with the histopathologic diagnoses. In total, 4,451 cases were analyzed. Dermatologists diagnosed twice the number of neoplastic and cystic skin lesions correctly (75%) than nondermatologists (40%). The clinical diagnosis rendered by family practitioners matched the histopathologic diagnosis in 26% of neoplastic and cystic skin lesions. Inflammatory skin diseases were correctly diagnosed in 71% of the cases by dermatologists but 34% of the cases by nondermatologists.

2004   Diagnostic Accuracy and Image Quality Using a Digital Camera for Teledermatology

The study was designed to evaluate the effectiveness of digital photography for dermatologic diagnoses and compare it with in-person diagnoses. There was 83% concordance [agreement] between in-person versus digital photo diagnoses. Concordance with biopsy results [agreement about accuracy] was achieved in 76% of the cases. Image sharpness and color quality were rated "good" to "excellent" 83% and 93% of the time, respectively.

2008   Diagnostic accuracy and appropriateness of general practitioner referrals to a dermatology out-patient clinic

A study was undertaken of new referrals by GPs to a dermatology clinic in a district general hospital over a 6-month period. 686 consecutive referrals to one consultant were analyzed for diagnostic accuracy and requirement for referral. 47% of referral letters contained the correct diagnosis. Viral warts and psoriasis were best diagnosed (82 and 78%, respectively). Seborrhoeic warts and dermatofibromas caused difficulty (22 and 19%, respectively). Cutaneous malignancy was correctly diagnosed in 45% of referrals, and eczema, the commonest condition referred, in 54% of cases.

                                                                                                                                                                           

2009    Teledermatology: A Review of Reliability and Accuracy of Diagnosis and Management

Accuracy rates ranged from 30% to 92%for clinic dermatologists [meaning in person] and from 19% to 95% for tele-dermatologists [meaning diagnosis via picture image].

Tele-dermatologists and clinic dermatologists completely agreed with each other in 41% to 94% of cases. They had partial agreement in 50% to 100%.

2012    Accuracy in skin cancer diagnosis: A retrospective study of an Australian public hospital dermatology department

Histology [under the microscope diagnosis] for all skin biopsies and excisions performed in an 18-month period at a public hospital dermatology department were reviewed. 6,546 biopsies/excisions were performed, identifying 55 melanomas. The sensitivity [also often thought of as accuracy] of melanoma diagnosis was 76%. 11% of melanomas were diagnosed as dysplastic naevi (moles).

There is a range in the level of accuracy. It is not surprising that dermatologists, who have more experience with skin have higher accuracy rates. It is interesting to note that accuracy is not 100% for dermatologists all of the time.

 

With regard to a skin image database another questions arises: Are 10,000 images that are 99% accurate better or worse than 1,000,000,000 images that are 60% accurate? 

At this point I think it is still an open question.