Questions and Answers: Part 2

Continued...

How will people find the site?

There are two different audiences for whom the site will hopefully be useful or interesting. One is a non-professional audience. This is the main group for whom the project is created. This group encompasses the broadest possible swath of the population. Initially the site will be promoted through word of mouth and personal social networks. At this point there is no budget for advertising. There will not be the ability to promote the site on search engines.

What is available at the beginning is a relatively small network. An initial marketing strategy will need to include free sources such as articles in local news/periodicals. Web based articles on relevant topics can be commented on and a link posted in the comments section. Connections with prominent individuals interested in the open-source arena and we can seek promotion of the project through their broader networks. As mentioned in the “who will use this” section – it will more likely be younger individuals, and venues such as Hacker News can be sought after to to promote the project to that demographic.

Who are the users of the image database?

Anyone. I have read many stories that describe products being used in completely different ways than the inventor originally imagined. I think the project, if viable, in and of itself demonstrates the promise of broader citizen participation in medical science. The original spark for the idea developed out of a conversation between a medical student and a computer scientist. That combination are potential users of the image database. 

I can imagine a clinical physician or researcher in an academic medical center using the large data files for academic projects or research. 

A database that can be displayed in an online webpage format – like the results of a Google image search – will have broad uses. I have watched as a competent doctor stepped out of the patient room and used a Google image search to compare what she saw to image search results during patient care - this database could provide, I believe, an equivalent result, without the extraneous results you get with that type of search. I have personally been a medical student looking at similar but smaller databases for examples of a rash while trying to learn - this could database could provide that with likely a larger volume of images. My hope is that be making the source code (the images) open, the project is not making a judgment on who can or cannot use the information or for what purpose. Rather, by providing aggregated information that is not currently available, it may create a category of users that I cannot yet imagine. My hope is to attract users that are currently beyond my imagination.

What kind of access do they need? 

Full and open access. This is a project to aggregate and to curate. A librarian does not own the knowledge in a library, but rather acts as a facilitator to others get at the information. This project is intended to be used in a similar way, as a facilitator to get at a particular type of information. With regards to the distribution and format of the database, I view this as partially a budget question. Initially, I envision a database of images (such as .jpg files) with metadata attached (such as age, gender) to it that are stored on a secure server. During the initial phases, access to the database can be requested via an email, and will be shared in a zipped file. This can be emailed or shared via online file sharing mechanism to anyone who requests access. It would be one of the least expensive ways to begin development.

A subsequent project is the ability to deliver the collected images online, in a manner visually similar to the results of a Google image search. This would allow for browsing/searchable access without the need to know how to manipulate the contents of a large zipped file. This will ability is more costly from a web development perspective, and unlikely to be completed in the initial phases of the project.

Are there any partnerships that you make ahead of time or will they be organic from the site? 

Partnerships will be multi-faceted. Some will be cultivated during the development, while it is hoped others will be created organically from the project itself. During the development phase partnerships from special groups may provide mutual benefit. Foundations that focus on a specific disease category such as psoriasis may find this project to add value to their members as a way to contribute to medical science. As such, they may seek to promote it among their membership. Additionally, non-medical groups such technologists passionate about open-source technology development may find the project interesting from their perspective and can promote the spread by word of mouth in order to participate in open-source health projects. Currently, there are few such projects.

How will user-submitted images/privacy be protected?

This is a critical question. In a previous post we discussed how people share photos online by the billions. I believe there is a large portion of society that is willing and desirous to spend a small amount of their time in order to contribute to a project that will have an aggregate good. At the same time, there is an imperative to protect personal/private information such as your personal health information.

Initial measures to protect privacy:

- Image suppliers will be encouraged not include pictures that contain the full of their face, tattoos or other easy to identify markings. 

- During uploading there will not initially be a username or log in, so as not to link a user with an image.

- The user will have the right to their own information, meaning they can request to have it deleted from the database at any time.

- We will not store any personal identifying information (PII) with the uploaded image, or even request it for uploading. The exception will be an email address, which will not be mandatory. An email address can provide a way to communicate with users about the project. Additionally, an email address can allow the user to request an image be deleted at some later date. 

- IP addresses will not be recorded at the time of image submission.

- No PII will be included in the distribution of the database.

-All privacy policies will be kept transparent, and will be reviewed periodically, by user request or suggestion, or if new information regarding privacy maintenance comes to light.

If anyone can download an image what protections, if any, are in place to protect the original submitters’ privacy? 

The primary goal is to provide an open space where this information is available. By adhering to the privacy strategy mentioned in the question above it is creating a layer of protection for the original submitter. A submission will constitute an implicit and explicit agreement to participate in the project and its purpose. By not asking for a name, address, credit card, or phone number there will be data to track the image to a certain individual. In order to request for your personal picture to be removed at any time there are several strategies to do that. One is to provide an email or to provide a unique tracking number for the image so it can be requested to be removed at any time. The database will not include any of this linking information. This type of privacy of, of a skin photo, is currently maintained in these databases that are actively available. In summary, the primary means of ensuring user privacy is to collect/store minimal amounts of personal information with the submitted image.

How will this project be sustained? 

At this point, there is no money generating from the project. At the initial phases it will be sustained with donations and grants and personal income.

There are several models of financial sustainability such as user fees, subscriptions, advertising, partnerships that find the project valuable to fund – however nothing specific has been decided at this point.

The Shuttleworth flash-grant is the reason the project will exist in the world beyond sketches on the back of a napkin.

Is there a commercial aspect to it? 

There is no commercial aspect to the project currently. While their may exist commercial applications in the future – the idea was conceived envisioning the ethos of Wikipeda, Ubuntu, and Ushahidi. These are all projects that provide value to those use them. They create value but are not specifically designed for the purpose of revenue generation. There are commercial models that could be applied to this type of platform such as advertising physician/dermatology offices based on location. This type of model will not be pursued at the outset.

Are you actively seeking additional funding?

Not at the moment.

What are other potential funding sources? 

More funding will not be pursued from outside sources until a proof of concept can be illustrated. A proof of concept means a working website, with ability to upload images to a database, and a database that includes pictures of skin with information attached. 

We first have to prove there is something of value worth funding in the idea. Until that point the small grant and out-of-pocket finances will provide the financial input. In the future, I have considered seeking funding from one of the crowd-sourced funding platforms such as Kickstarter, Indiegogo, or foundations.

Are you supporting this out of pocket as a labor of love? 

We have received one grant but this is also a labor of love.

Questions and Answers: Part 1

One of the underlying assumptions about the crowd-sourced skin image database is that an open platform to gather information can produce as good, or better, results. The open-ness of the platform allows for non-professionals to contribute in a way that will add value.

Some details of the project have been described elsewhere in this blog. Below is an opportunity to discuss some specific questions about the project.

All of the questions and answers are directly related to the idea of a Crowd-sourced dermatology database. The project will be a web (and eventually mobile) based platform for individuals to upload images of a skin lesion or condition that has previously been evaluated and diagnosed by a medical professional. Requiring a diagnosis from a medical professional is for the purpose of facilitating greater accuracy. Doctors see millions of patients per year with skin related issues. Small percentages of those daily encounters are currently being captured in a database for future use. The intention is Not for self-diagnosis by patients, but rather to enable individuals to contribute to the scientific/biological body of knowledge. The images, freely given to the database, will be de-identified. The privacy of an individual’s health/skin related concern is as important as the collection of skin images for furtherance of medical science. The collection of images has value in the form of aggregated biological information. The database will be open-source. One way to view this project is thus: skin rashes/skin diseases are a manifestation of our common biological and genetic code. By aggregating submitted skin pictures from a global community into database, and then sharing that biologic information in an open source environment starts to create for medical science what computer science has done with open-sourcecomputer code.

_________________________________________________________________________

Question & Answer

Who are your target users? 

This is a marketing question. This frames the issue in terms of narrowing to a unique user group. Narrowing to a specific a group thus makes the problem of spreading the word about the project easier. No specific category or group will intentionally be excluded from the project.

Millions of individuals in the US are seen yearly in medical facilities for a skin condition. There are more globally, although that data is harder to find. A percentage of these individuals who seek professional medical care for a dermatologic issue will be willing to contribute a picture (taken by themselves or others) of their skin for a project to contribute to the greater scientific/medical good. The greater good in this case is the aggregate of many people’s skin images. A target user has access to an internet connection (via mobile or desktop) and access to camera. Increasingly, that is through a smartphone camera with web access. The target user is more likely younger (15-35 demographic) given the facility and comfort with sharing information online. A user/contributor will likely have a strong interest in their personal health and an interest in contributing what they have learned about themselves to a larger project that has endurance beyond that moment in time. The individual action contributes a single potential to a greater good. A contribution is an individual action for others to use, however not in the exact same way as Wikipedia. A survey of Wikipedia users and contributors suggested that it was primarily young (26yr), single (~67%), males (86%) who had completed high or college (33% and 25% respectively) that contributed articles to the website.

 Individuals who have an interest in health research are also target users. Those who have desire to utilize an open database will also have incentive to promote image contributions to the site. Target users with a specific interest in the database include health educators (to use the site to show an array of images of a certain type) and learners (medical students, nurses).

Who are the people who will submit pictures?

Anyone willing to contribute a picture of an abnormal skin lesion with a diagnosis, which has the rights to do so, is welcome. The expectation is that the image has been seen by a medical professional such as a nurse practitioner or an emergency room physician and that diagnosis will be linked to the submitted image. Two broad categories of people are envisioned as image submitters: Non-professionals and medical professionals. A non-professional means a non-medical professional. A bank teller, a lawyer, a student are all examples. The non-professional is envisioned as submitting an image of themselves, their own individual skin. At the point of image submission it will be explicit in the fact that personal information will not be linked in order to protect the personal medical issues of those wishing to contribute. Image submissions from non-professionals other than the primary individual (such as a friend or relative) will be allowed, but made explicit the person whose skin is in the picture has given consent for that image to be shared in a de-identified manner.

Medical professionals may also contribute images. A health practitioner with the interest, time, and inclination, with the patients’ explicit consent, can submit images.

Why do they want to upload their picture?

At the initial stages, the immediate gratification for the contribution will be a thank you email (if an email address has been provided) and the knowledge and warmth that comes from contributing to the greater good. Why do people contribute to online forums for car maintenance, or pot-hole repair, or Wikipedia articles? There is a satisfaction that comes from contributing to something larger than yourself.

The psychology of this contribution model will use that as an underlying assumption: people are inherently good. People desire to be part of a project for a greater good. This is not to say that we act in altruistic and benevolent ways in all our every action. The main goal for initial iteration would be building up a database of images. People interested in contributing to that project will upload their picture. Even at a fraction of the potential user base, the will create a much larger open-source dermatology database than currently exits. That is because the number of images in any open-source dermatology database = 0.

There are several ideas that are being considered for user feedback that will help motivate and keep contributors engaged/active/and spreading the word.

Possible incentive structures for an image contribution:

• Access to images with similar diagnosis – allowing a user to browse an online picture bank of images with similar diagnosis “oh that looks like what my skin looks like…”
• Allowing users who submit images to track changes in lesions or moles over time. A picture taken year on year could provide side by side comparison for changes in size/shape/color. This would require the ability to match the image to the same individual, such as a user account. 
• Allowing specialized dermatology interest groups with dispersed geography to share images, as the database will be openly available to them.
• Suggestions are welcome.

Who is making the diagnosis?

This has been discussed several times, including in this post above.

A diagnosis will be made by a medical professional. The credibility of the medical professional is implicit in the fact of image submission. It will be articulated at the time of image submission, then implicitly assumed that user has entrusted their skin issue to someone with the ability to evaluate and manage skin conditions. It is not the aim of the project to verify medical licenses. In addition to submitting the image and the diagnosis, the user will be asked to provide several pieces of additional information. One piece of information will be the type of medical professional making the diagnosis. Professionals can include: Nurse Practitioner (NP), Doctor, Physician’s Assistant (PA), Dermatologist, Family Doctor, Emergency Room Physician.

Who are the people reviewing the pictures? 

At the outset, when image volumes are low, I will be reviewing the pictures. This will not be a diagnostic review. The initial image review will focus on visibility/clarity of the image. The image review will assess if it is a picture of skin, or is it not skin. It will look for any identifying features in the photo such as names or faces. Images that do not pass this bar will be sent back to the submitter and not included in the database. The number of images submitted will be counted in order to determine the percentage that are submitted but are not appropriate (such as a non-skin image). Once the image volume becomes great to be handled by a single individual others will be asked to assist and divide up the work. The bio and contact information of anyone with access to the data of images will be provided transparently on the site. In subsequent iterations it is one of the goals to have users contribute to the review process. It does not take a medical professional to look at a series of images and determine which a picture of skin and which is a picture of Mickey Mouse. Non-professionals can also review images and flag them for image clarity.

Accuracy

Accuracy is an interesting thing in medicine. There is an assumption, this unspoken hope, from those who seek medical care that the answer they get is correct. Certainty breeds confidence, and confidence (without reference to accuracy) breeds a measure of certainty. A diagnosis, confidently stated can lead to a feeling of reassurance, and that the answer/the diagnosis is accurate. Physicians are intensely trained, and have a high level of professionally acquired skill. However, medical practitioners are not perfect. The idea that medical purveyors are inacurate, and do not come up with the correct answer the first time (or the second or third time) propels the success of television dramas like House, M.D.

When thinking about curating a medical database of many user submitted images, the topic of accuracy arises. Can we trust a citizen, a user, a non-professional to contribute an image of a skin lesion? Will it be labeled with the correct diagnosis? In my mind this is an open question. As described previously, we like to share images - think about the number of pictures uploaded to Facebook daily. The number of encounters with medical professionals related to skin issues far outstrips the number of pictures in currently available databases (see prior posts). It is the belief of this project that people can be trusted to contribute to a scientific project in a meaningful (meaning accurate) way. As a first pass, the uploaded image will rely on the diagnostic accuracy of the medical professional who evaluates the rash or lesion. It is trust given to the user/image contributor that they are contributing what a medical professional has communicated.

The question that has been in my mind recently – How accurate are medical professionals when pronouncing a diagnosis regarding the skin?

I conducted a review in the published scientific literature and come across some information shared below which will shed some light on this question. This is by no means an exhaustive review. If you are interested in the full paper please email me, however they are easily accessible via Pubmed. Below you will find a listing of medical/scientific studies from the 1970s through to the present which look into this question. The year of the study publication is listed as well as the title of the medical paper and a brief description along with an accuracy rate in the text. In many cases accuracy is defined as the diagnosis given to the picture or patient after examination as compared to the correct diagnosis which is typically a biopsy-which serves as the final arbiter of correctness in skin diagnosis.

1972    Accuracy of Dermatologic Diagnosis by Television

Comparisons of diagnoses made by black and white television viewing of slides of dermatologic lesions to those made from directly viewing of the same slides in color revealed that in 85% to 89% of cases the dermatologists were as accurate by television as on direct examination. Color television improved accuracy only slightly, but was more acceptable to the dermatologists as less time was required to reach a diagnosis.

1983    The Prevalence and Accuracy of Diagnosis of Non-Melanotic Skin Cancer in Victoria

Surveys of Victorian [Australian] dermatologists and pathologists were undertaken to determine the number of patients attending medical practitioners with non-melanotic [non-cancer] skin cancers and solar keratoses. Accuracy of clinical diagnosis studies suggest that the correct diagnosis of these tumors is being made clinically [in person] in approximately 70% of cases by experienced clinicians.

1989    The Development of Expertise in Dermatology

To examine the development of expertise in dermatology at five levels of expertise. A total of 100 slides [pictures], 2 typical and 3 atypical, from each of 20 common skin disorders, were presented to six subjects at each of the following levels: second-year preclinical medical students, final year medical students, residents in family medicine, general practitioners, and dermatologists. Accuracy of diagnosis rose from 21% for medical students to 87% for dermatologists.

1990    Accuracy in the Clinical Diagnosis of Malignant Melanoma

The computerized database (1955 through 1982) of the Oncology Section of the Skin and Cancer Unit of New York (NY) University Medical Center includes data on 13,878 lesions. Of these lesions, 214 were diagnosed clinically and histologically as malignant melanoma (MM)...The diagnostic accuracy for the best period (1974 - 1982) was 64%. The diagnosis of MM was made in 84.5% of the histologically proved cases of MM, reflecting a high degree of sensitivity.

 

2001    A comparison of dermatologists' and primary care physicians' accuracy in diagnosing melanoma: a systematic review

Studies were evaluated to determine the sensitivity and specificity of dermatologists' or PCPs' [Primary Care Providers - also known as general practitioners, commonly family doctors or internal medicine doctors in the US] ability to correctly diagnose lesions suggestive of melanoma and to perform biopsies or refer patients with such lesions. For diagnostic accuracy, sensitivity was 0.81 to 1.00 for dermatologists and 0.42 to 1.00 for PCPs. None of the studies reported specificity for dermatologists; one reported specificity for PCPs (0.98). For biopsy or referral accuracy, sensitivity ranged from 0.82 to 1.00 for dermatologists and 0.70 to 0.88 for PCPs; specificity, 0.70 to 0.89 for dermatologists and 0.70 to 0.87 for PCPs. Receiver operating characteristic curves for biopsy or referral ability were inconclusive. The study concluded that published data are inadequate to demonstrate differences in dermatologists' and PCPs' diagnostic and biopsy or referral accuracy of lesions suggestive of melanoma.

2003    Comparison of diagnostic accuracy for cutaneous malignant melanoma between general dermatology, plastic surgery and pigmented lesion clinics.

Since the 1980s there have been dedicated pigmented lesion clinics (PLCs) in the U.K. This study compared the false-negative rate (FNR) of clinical diagnosis with other clinics of primary referral of malignant melanoma (MM) in the same geographical area.

The case notes of 731 patients were available, of whom approximately two-thirds initially attended the PLC, one-fifth the General Dermatology clinics (D) and the remainder were divided approximately between Plastic Surgery clinics (P), other clinics (O) and the general practitioner (GP). The FNR was lowest for the PLC, at 10%, compared with 29% (D), 19% (P), 55% (O) and 54% (GP) (P < 0.0001).

**Accuraccy rates can be considered 100 - percentage above (ie: Accuracy at the dermatology clinic: 100 - 29% false negatives = 71% accurate/true positive diagnosis.

2003    Pattern analysis, not simplified algorithms, is the most reliable method for teaching dermoscopy for melanoma diagnosis to residents in dermatology

This study investigated the diagnostic performance of three different methods of teaching dermoscopy when used by newly trained residents in dermatology to diagnose melanocytic [cancerous] lesions. Pattern analysis yielded the best mean diagnostic accuracy (68.7%), followed by the ABCD rule (56.1%) and the seven-point check-list (53.4%, P = 0·06).

2004   A retrospective biopsy study of the clinical diagnostic accuracy of common skin diseases by different specialties compared with dermatology

The clinical diagnoses of family physicians, plastic, general, and orthopedic surgeons, and internists and pediatricians versus dermatologists were correlated with the histopathologic diagnoses. In total, 4,451 cases were analyzed. Dermatologists diagnosed twice the number of neoplastic and cystic skin lesions correctly (75%) than nondermatologists (40%). The clinical diagnosis rendered by family practitioners matched the histopathologic diagnosis in 26% of neoplastic and cystic skin lesions. Inflammatory skin diseases were correctly diagnosed in 71% of the cases by dermatologists but 34% of the cases by nondermatologists.

2004   Diagnostic Accuracy and Image Quality Using a Digital Camera for Teledermatology

The study was designed to evaluate the effectiveness of digital photography for dermatologic diagnoses and compare it with in-person diagnoses. There was 83% concordance [agreement] between in-person versus digital photo diagnoses. Concordance with biopsy results [agreement about accuracy] was achieved in 76% of the cases. Image sharpness and color quality were rated "good" to "excellent" 83% and 93% of the time, respectively.

2008   Diagnostic accuracy and appropriateness of general practitioner referrals to a dermatology out-patient clinic

A study was undertaken of new referrals by GPs to a dermatology clinic in a district general hospital over a 6-month period. 686 consecutive referrals to one consultant were analyzed for diagnostic accuracy and requirement for referral. 47% of referral letters contained the correct diagnosis. Viral warts and psoriasis were best diagnosed (82 and 78%, respectively). Seborrhoeic warts and dermatofibromas caused difficulty (22 and 19%, respectively). Cutaneous malignancy was correctly diagnosed in 45% of referrals, and eczema, the commonest condition referred, in 54% of cases.

                                                                                                                                                                           

2009    Teledermatology: A Review of Reliability and Accuracy of Diagnosis and Management

Accuracy rates ranged from 30% to 92%for clinic dermatologists [meaning in person] and from 19% to 95% for tele-dermatologists [meaning diagnosis via picture image].

Tele-dermatologists and clinic dermatologists completely agreed with each other in 41% to 94% of cases. They had partial agreement in 50% to 100%.

2012    Accuracy in skin cancer diagnosis: A retrospective study of an Australian public hospital dermatology department

Histology [under the microscope diagnosis] for all skin biopsies and excisions performed in an 18-month period at a public hospital dermatology department were reviewed. 6,546 biopsies/excisions were performed, identifying 55 melanomas. The sensitivity [also often thought of as accuracy] of melanoma diagnosis was 76%. 11% of melanomas were diagnosed as dysplastic naevi (moles).

There is a range in the level of accuracy. It is not surprising that dermatologists, who have more experience with skin have higher accuracy rates. It is interesting to note that accuracy is not 100% for dermatologists all of the time.

 

With regard to a skin image database another questions arises: Are 10,000 images that are 99% accurate better or worse than 1,000,000,000 images that are 60% accurate? 

At this point I think it is still an open question.