Accuracy

Accuracy is an interesting thing in medicine. There is an assumption, this unspoken hope, from those who seek medical care that the answer they get is correct. Certainty breeds confidence, and confidence (without reference to accuracy) breeds a measure of certainty. A diagnosis, confidently stated can lead to a feeling of reassurance, and that the answer/the diagnosis is accurate. Physicians are intensely trained, and have a high level of professionally acquired skill. However, medical practitioners are not perfect. The idea that medical purveyors are inacurate, and do not come up with the correct answer the first time (or the second or third time) propels the success of television dramas like House, M.D.

When thinking about curating a medical database of many user submitted images, the topic of accuracy arises. Can we trust a citizen, a user, a non-professional to contribute an image of a skin lesion? Will it be labeled with the correct diagnosis? In my mind this is an open question. As described previously, we like to share images - think about the number of pictures uploaded to Facebook daily. The number of encounters with medical professionals related to skin issues far outstrips the number of pictures in currently available databases (see prior posts). It is the belief of this project that people can be trusted to contribute to a scientific project in a meaningful (meaning accurate) way. As a first pass, the uploaded image will rely on the diagnostic accuracy of the medical professional who evaluates the rash or lesion. It is trust given to the user/image contributor that they are contributing what a medical professional has communicated.

The question that has been in my mind recently – How accurate are medical professionals when pronouncing a diagnosis regarding the skin?

I conducted a review in the published scientific literature and come across some information shared below which will shed some light on this question. This is by no means an exhaustive review. If you are interested in the full paper please email me, however they are easily accessible via Pubmed. Below you will find a listing of medical/scientific studies from the 1970s through to the present which look into this question. The year of the study publication is listed as well as the title of the medical paper and a brief description along with an accuracy rate in the text. In many cases accuracy is defined as the diagnosis given to the picture or patient after examination as compared to the correct diagnosis which is typically a biopsy-which serves as the final arbiter of correctness in skin diagnosis.

1972    Accuracy of Dermatologic Diagnosis by Television

Comparisons of diagnoses made by black and white television viewing of slides of dermatologic lesions to those made from directly viewing of the same slides in color revealed that in 85% to 89% of cases the dermatologists were as accurate by television as on direct examination. Color television improved accuracy only slightly, but was more acceptable to the dermatologists as less time was required to reach a diagnosis.

1983    The Prevalence and Accuracy of Diagnosis of Non-Melanotic Skin Cancer in Victoria

Surveys of Victorian [Australian] dermatologists and pathologists were undertaken to determine the number of patients attending medical practitioners with non-melanotic [non-cancer] skin cancers and solar keratoses. Accuracy of clinical diagnosis studies suggest that the correct diagnosis of these tumors is being made clinically [in person] in approximately 70% of cases by experienced clinicians.

1989    The Development of Expertise in Dermatology

To examine the development of expertise in dermatology at five levels of expertise. A total of 100 slides [pictures], 2 typical and 3 atypical, from each of 20 common skin disorders, were presented to six subjects at each of the following levels: second-year preclinical medical students, final year medical students, residents in family medicine, general practitioners, and dermatologists. Accuracy of diagnosis rose from 21% for medical students to 87% for dermatologists.

1990    Accuracy in the Clinical Diagnosis of Malignant Melanoma

The computerized database (1955 through 1982) of the Oncology Section of the Skin and Cancer Unit of New York (NY) University Medical Center includes data on 13,878 lesions. Of these lesions, 214 were diagnosed clinically and histologically as malignant melanoma (MM)...The diagnostic accuracy for the best period (1974 - 1982) was 64%. The diagnosis of MM was made in 84.5% of the histologically proved cases of MM, reflecting a high degree of sensitivity.

 

2001    A comparison of dermatologists' and primary care physicians' accuracy in diagnosing melanoma: a systematic review

Studies were evaluated to determine the sensitivity and specificity of dermatologists' or PCPs' [Primary Care Providers - also known as general practitioners, commonly family doctors or internal medicine doctors in the US] ability to correctly diagnose lesions suggestive of melanoma and to perform biopsies or refer patients with such lesions. For diagnostic accuracy, sensitivity was 0.81 to 1.00 for dermatologists and 0.42 to 1.00 for PCPs. None of the studies reported specificity for dermatologists; one reported specificity for PCPs (0.98). For biopsy or referral accuracy, sensitivity ranged from 0.82 to 1.00 for dermatologists and 0.70 to 0.88 for PCPs; specificity, 0.70 to 0.89 for dermatologists and 0.70 to 0.87 for PCPs. Receiver operating characteristic curves for biopsy or referral ability were inconclusive. The study concluded that published data are inadequate to demonstrate differences in dermatologists' and PCPs' diagnostic and biopsy or referral accuracy of lesions suggestive of melanoma.

2003    Comparison of diagnostic accuracy for cutaneous malignant melanoma between general dermatology, plastic surgery and pigmented lesion clinics.

Since the 1980s there have been dedicated pigmented lesion clinics (PLCs) in the U.K. This study compared the false-negative rate (FNR) of clinical diagnosis with other clinics of primary referral of malignant melanoma (MM) in the same geographical area.

The case notes of 731 patients were available, of whom approximately two-thirds initially attended the PLC, one-fifth the General Dermatology clinics (D) and the remainder were divided approximately between Plastic Surgery clinics (P), other clinics (O) and the general practitioner (GP). The FNR was lowest for the PLC, at 10%, compared with 29% (D), 19% (P), 55% (O) and 54% (GP) (P < 0.0001).

**Accuraccy rates can be considered 100 - percentage above (ie: Accuracy at the dermatology clinic: 100 - 29% false negatives = 71% accurate/true positive diagnosis.

2003    Pattern analysis, not simplified algorithms, is the most reliable method for teaching dermoscopy for melanoma diagnosis to residents in dermatology

This study investigated the diagnostic performance of three different methods of teaching dermoscopy when used by newly trained residents in dermatology to diagnose melanocytic [cancerous] lesions. Pattern analysis yielded the best mean diagnostic accuracy (68.7%), followed by the ABCD rule (56.1%) and the seven-point check-list (53.4%, P = 0·06).

2004   A retrospective biopsy study of the clinical diagnostic accuracy of common skin diseases by different specialties compared with dermatology

The clinical diagnoses of family physicians, plastic, general, and orthopedic surgeons, and internists and pediatricians versus dermatologists were correlated with the histopathologic diagnoses. In total, 4,451 cases were analyzed. Dermatologists diagnosed twice the number of neoplastic and cystic skin lesions correctly (75%) than nondermatologists (40%). The clinical diagnosis rendered by family practitioners matched the histopathologic diagnosis in 26% of neoplastic and cystic skin lesions. Inflammatory skin diseases were correctly diagnosed in 71% of the cases by dermatologists but 34% of the cases by nondermatologists.

2004   Diagnostic Accuracy and Image Quality Using a Digital Camera for Teledermatology

The study was designed to evaluate the effectiveness of digital photography for dermatologic diagnoses and compare it with in-person diagnoses. There was 83% concordance [agreement] between in-person versus digital photo diagnoses. Concordance with biopsy results [agreement about accuracy] was achieved in 76% of the cases. Image sharpness and color quality were rated "good" to "excellent" 83% and 93% of the time, respectively.

2008   Diagnostic accuracy and appropriateness of general practitioner referrals to a dermatology out-patient clinic

A study was undertaken of new referrals by GPs to a dermatology clinic in a district general hospital over a 6-month period. 686 consecutive referrals to one consultant were analyzed for diagnostic accuracy and requirement for referral. 47% of referral letters contained the correct diagnosis. Viral warts and psoriasis were best diagnosed (82 and 78%, respectively). Seborrhoeic warts and dermatofibromas caused difficulty (22 and 19%, respectively). Cutaneous malignancy was correctly diagnosed in 45% of referrals, and eczema, the commonest condition referred, in 54% of cases.

                                                                                                                                                                           

2009    Teledermatology: A Review of Reliability and Accuracy of Diagnosis and Management

Accuracy rates ranged from 30% to 92%for clinic dermatologists [meaning in person] and from 19% to 95% for tele-dermatologists [meaning diagnosis via picture image].

Tele-dermatologists and clinic dermatologists completely agreed with each other in 41% to 94% of cases. They had partial agreement in 50% to 100%.

2012    Accuracy in skin cancer diagnosis: A retrospective study of an Australian public hospital dermatology department

Histology [under the microscope diagnosis] for all skin biopsies and excisions performed in an 18-month period at a public hospital dermatology department were reviewed. 6,546 biopsies/excisions were performed, identifying 55 melanomas. The sensitivity [also often thought of as accuracy] of melanoma diagnosis was 76%. 11% of melanomas were diagnosed as dysplastic naevi (moles).

There is a range in the level of accuracy. It is not surprising that dermatologists, who have more experience with skin have higher accuracy rates. It is interesting to note that accuracy is not 100% for dermatologists all of the time.

 

With regard to a skin image database another questions arises: Are 10,000 images that are 99% accurate better or worse than 1,000,000,000 images that are 60% accurate? 

At this point I think it is still an open question. 

Informational Asymmetry

Accurate, timely, relevant information is a prerequisite for effective delivery of healthcare services at the individual and population level. A healthcare provider uses information, consumed in a myriad of forms, and filters the filters the new information through a framework of experience, to arrive at a conclusion and a plan of action. Jerome Groopman has detailed many of these thought patterns of doctors in How Doctors Think. The book describes numerous limitations to informational processing at the individual level.

An example clinical encounter is described:

A 28 year old female develops a painful rash on her left forearm. It starts without any apparent cause to the patient. The young lady has no other chronic medical issues and takes no regular medications. There has been no travel, no recent contact with any sick persons, and no change in her regular routine as a consultant. For several days the rash progressively worsens, and pain increases.  She schedules a visit with a dermatologist. In the medical encounter with the health professional a broad list of possibilities is considered. It could be one of several types of infections, it could be an allergic reaction, it could be underlying chronic skin condition that is just manifesting at this time. Tests of the skin are obtained, and treatment was prescribed.

Informational input needs to be accurate. The best thing in medicine is the people, the patients. Patients know the most about themselves and how they are feeling at any particular time. The challenge is in the details. For those not familiar with medical language it is not easy to remember the specific terminology, or names of medications. Was it ciprofloxacin or levofloxacin? What was the result of the heart test? Was there any particular exposure to the skin, or any new detergent, or perfume? Many times details of exposure and chronology stand out in peoples’ minds, often times they do not.

At the level of the professional, clinical information is filtered through an information mesh-work to pull out the important pieces of data in order to arrive at the conclusion/diagnosis. The process of training physicians in the U.S. system involves a high volume of exposure to individual patient care in an appropriately graduated structure. The broad experience base generates a mental framework that subsequent information is processed through. This is what Groopman discusses as heuristics; the experience-based technique for problem solving. The challenge in medical training is that is often hard to infuse more wisdom than their experience allows. Certainly, information acquired through reading builds and expounds the mental framework. That is how a medical student can know someone has appendicitis before they have ever seen or diagnosed their first case. The modern challenge is that there are now (as of 2010) 75 medical trials, and 11 systematic reviews of trials, per day.(Ref) All of that newly created information adds layers of nuance to the diagnostic and treatment processes. The human body and its ailments have not suddenly altered, but the highest value testing and treatment practices do change over time. This is the modern challenge, the smartest individual in the world does not have the capacity to consume and retain all of that information. An internet connected provider, speaks to a potential of leveling the playing field, by having access to up to date information. This ultimately will be a good thing. 

Asymmetry is the disunion that happens when the processing framework fails to properly connect and add value during the throughput of individual data. Miss-connections include:  the drug was X, not Y as the patient had originally remembered it. The previous electrocardiogram (EKG) was not available to compare for any change. From the provider perspective: that type of case, with similar type symptoms, lab results, and outcome has not been encountered before. Is this a new presentation of a new disease, or an unusual combination of symptoms for a common disease. Or from a dermatology perspective - a specific type of skin rash has not been seen before.

Facilitating a more open posture for informational flow, such as producing open source medical databases, offers a potential to augment the processing framework. At the individual level an open-source dermatology database (adequately marketed) could be searched, just like I have observed Attending physicians search Google Images for skin diagnosis. The open data could be accessed to augment the providers information mesh-work in order to decrease the probability of asymmetry. The leverage of computers/screens/phones is informational connection. This rationale assumes that the answer is out there somewhere. Finding the answer is a matter of reducing the asymmetry. When we open informational networks it is not immediately clear how commercial value will be derived. However, connecting people to the right information at the right time is one of the challenges we have taken on as we try and keep ourselves and our society healthy

The young lady described above is my wife. After time, testing, and contact with multiple medical professionals, she was ultimately diagnosed with phyto-photo dermatitis. That term was coined by Klaber in 1942.(Ref) It is a skin eruption caused by exposure to certain plants and their extracts and then being exposed to sunlight. Kelly reviewed the literature and reported that various authors make mention of the fact that the condition was probably known of in countries such as India, Arabia and Egypt many centuries before Christ.(Ref) The rash resolves over time, and with some steroid cream. She is doing well. Reducing the asymmetry in our own lives would have helped us avoid such an arduous process for a rash that has been around and known for such a long time. 

Forward Movement

I recently received an email, the contents of which are shared below:

CONGRATULATIONS!

You have been nominated by Andrew Rens to receive a Shuttleworth Foundation Flash Grant to work on your CrowdSourced Dermatologic Picture Database. We have a funding model that rewards brilliance and new ideas by awarding full-time Fellowships. However, bubbling around
the periphery of our model it has become clear that there are voices that are not heard, ideas not seen and Fellows that are not ready to be Fellows - just yet... This is where you and the new Shuttleworth Flash Grant come in.

We tasked our Fellows to seek out and nominate an impressive change agent, who may not be able to concentrate on their brilliant idea yet. This grant is for the sum of $5,000 and will be awarded to you personally to bring that idea forward. The only string attached is that we ask you to live openly, tell us and the world what you have done with the money.

_______________________________________________________________________

I am very thankful for the opportunity. This pilot financing will be used for website development and beta-launch. The bottleneck up to this point has been mobile and web development. The goal is to use as many open-source platforms as possible in order to limit cost and because the project speaks to open-source values.

Concept for Open Source Dermatology Database

The pilot grant will be used for the development of a platform (website-mobile if finances allow). The platform will be used to facilitate the proof of the following idea: An global online community of non-professionals can effectively create a database of dermatology skin pictures of sufficient accuracy and volume such that it will produce a database that is more valuable compared to current proprietary databases. One unique value of this project is that the product – the database with tagged skin images will be made available under a creative commons data license to all. The data represents a collection of global biological information. The project is trying to use the website as a platform for sharing this knowledge in the form of skin pictures.

This has never been tried before, in quite this way. Patientslikeme.com enables the sharing of patient contributed information, the angle is more from a patient support group perspective. Another uniqueness for this project is the idea of the medical community inviting wisdom and participation from patients. The openness and invitation to contribute is a different posture compared to the current status of medical knowledge creation. Medical research is currently conducted in such a way that information about how you reacted to a medication is recorded from you as a one in a large study. The information is stored in a data-set and is the property of the university medical center or pharmaceutical company conducting the research. The research subjects rarely have complete/open/free access that they were a part of creating. The current posture infuses the culture surrounding personal medical records in the United States. In many instances, there is formality and red tape to cut through in order to obtain your personal health information - in an age where access to information is becoming more ubiquitous. 

This concept brings up large questions. At this point, these are open questions for the which the implementation of the project will seek to address. That is a part of the joy of the project for me. From the viewpoint of the physician, one question is how accurate do you need the diagnosis on the submitted picture. We know that not every visit with a doctor generates the correct answer with regards to a skin rash. There are not the resources to follow up and send  a dermatologist and examine and biopsy every piece of skin from a submitted image. The first layer is to ask the user to submit an image of a skin lesion or rash only after it has been evaluated by a medical professional.

Several other open questions:

How do you ensure unhelpful image submissions are limited (ie. non skin)?

What is the photographic/pixel quality needed in order for the image to be seen clearly/re-usable and ultimately useful?

What data needs to be submitted along with the image? To much information and you lose participants, too few pieces of info (age, diagnosis) and you limit the power of the database.

How do you guard against fraud such as fake/false submissions?

Should you be required to submit an image in order to use the database?

At this point I am seeking to engage with web developers and move the idea forward.