The wisdom of crowds started in 1906 with Francis Galton predicting the weight of a dead ox. Since then it was successfully used in many fields as politics, weather, international affairs, sports and economics. Now we have what many would define as the most astonishing application of collective intelligence. CrowdMed aims to predict illness diagnoses from patients that have not been able to get an answer from the doctor.
We are passionate about prediction markets, so from Futura Markets we asked Jared Heyman, CrowdMed’ founder and CEO, who kindly accepted, to have a chat on collective intelligence and explain their experience in the heathcare system.
Jared Heyman, tell us that “average patient has seen 8 doctors and been sick for 6 years by the time they submit their case”. This is quite the ideal landscape for prediction markets to work: diverse information, uncertainty and market incentives to reveal the best pieces of information as best aggregation tool. Letting people bet on the most probable diagnosis for the symptoms submitted allows CrowdMed to separate the signal from the noise, using markets as a filter.
One of the most well known pieces of advice from successful entrepreneurs is “build something you would like to use”. This is exactly what Jared did. His young sister was had been ill for more than 3 years, and had to spend more than $100k in medical expenses and see dozens of doctors till she got the correct diagnosis, a rare genetic disorder. Jared tested CrowdMed with his sister’ case, and guess what:
“Through crowdsourcing, the participants were able to identify in two weeks what took doctors more than three years to diagnose”
When you think about disruptive innovation, with no doubt this is a perfect candidate when talking about the healthcare industry. By the way, CrowdMed has raised so far $2,4 million, being one opf the latest participant in the prestigious YCombinator, and including between his investor entities like NEA, Andreessen Horowitz, Greylock Partners, and SV Angel.
* You can check the spanish version of this interview here *
:: Should we say the «wisdom of crowds» or the «wisdom of markets»? Where is it the predictive power, if any?
Prediction markets as just one method (of many) for harnessing the wisdom of crowds. For CrowdMed’s purposes, we believe it to be the most powerful. Prediction markets have been used over the years to predict a variety of future outcomes with amazing predictive accuracy. In our case, we’re predicting “What’s most likely to be the correct diagnosis for this patient?”
:: Nate Silver tell us in his book «The Signal and the Noise» that every piece of information is decomposed in signal (truth) and noise (randomness). But medicine is a very specialized field of study. Is there any signal to aggregate from people apart from doctors? Which kind of users do you have?
Separating signal and noise is the hardest part of what we do technically at CrowdMed. Within our crowd of Medical Detectives, there are individuals who know the right diagnosis for a given patient (signal) and individuals who do not know or have the wrong answer (noise). CrowdMed’s job is to separate signal and noise, and pass along the signal to the patient. We believe strongly that the signal may not come from a doctor — the insight that leads a patient closer to their correct diagnosis and treatment could come from anywhere. About 60% of our Medical Detectives either work in or study medicine, and the other 40% are laypeople (often other patients who have experienced the same illness). We like having this mix between medical professionals and non-medical professionals, because both groups have useful information to share.
:: Robin Hanson has asked himself the question «why bets do look bad?» And, if bets have bad press, the situation will surely be worse when applied to medicine, being even shocking to outsiders. Seeing people betting on somebody’s illness is somehow counterintuitive. How do you explain the potential benefits of betting to new comers in CrowdMed?
There is something about “betting” that many societies find distasteful, perhaps because it’s associated with gambling, which is considered a sinful business. But prediction markets, by their nature, require betting in order to function properly. So, we had to balance the societal bias against betting with the advantages that prediction markets can bring to medical diagnosis. Our solution was to downplay the betting and “stock market” aspects of our site as much as possible to our users. So, while our Medical Detectives do place point bets on what they believe to the best answer, they never bet their own money, we never make reference to stock prices or show them a stock market graph, and they never bet on whether a morbid outcome will occur. We explain in our FAQs (https://www.crowdmed.com/faqs) why we use a point betting system.
:: The procedure to crowdsource an illness works like this: patients tell their symptons and medical detectives upload new possible diagnosis or bet on current choices. But how do you resolve the questions? Are patients responsible for choosing the correct answers?
Patients are instructed to consult with their physician and then report back to us with the best diagnostic and solution suggestions they received from our Medical Detectives. We allow them 30 days post-case to do so.
:: What it may sometimes happen is that the patient gets the diagnosis first, and needs a second opinion. In this case, do you allow people betting only on a particular diagnosis?
If a patient has a diagnosis and needs a second opinion, they would post their case to our site the same as anyone else. We never limit the diagnoses that can be suggested and bet upon.
:: Crowdsourcing some sensitive matters like your illness may break the medical rule «first do not harm». When betting, people take risks … including failing on the diagnosis. Do you have any control on this?
Our average patient has seen 8 doctors and been sick for 6 years by the time they submit their case on our site. They have already exhausted their options in the traditional medical system and need an alternative source of answers. Even if our Medical Detective community can’t produce any better diagnostic or solution suggestions than the patient’s physicians, they have little to lose and much to gain by trying CrowdMed.
:: However … Healthcare system is one of the best examples of «information assimetries». You cannot know whether the doctor is telling you the best solution for your illness or for his own pocket (that’s why the Hippocratic Oath exists). Do markets’ incentives allign the interest of the medical detective to the interest of the patient?
There’s a certain purity to our site in that our patients’ and Medical Detectives’ interests are perfectly aligned. The Medical Detectives’ only incentive is to produce the best diagnostic and solution suggestions for the patient.
:: Prediction markets is about two things: obtaining good estimates about future events, and classifying predictors. Could you please tell us your more impressive results in both aspects. Any outstanding Medical Detective? What’s the accuracy rate of your predictions?
Obtaining the best medical suggestions for our patients and classifying the performance of our Medical Detectives are both achieved on our site. In roughly 80% of cases, our patients tell us that the crowd produced accurate diagnostic suggestions. One can easily identify our most outstanding Medical Detectives via our leaderboards.
:: Google Flu Trends suffered tough critiques for failing on predicting last year epidemic trends. On the other hand, we have good results of experiments with prediction markets estimating the evolution of infectious deseases. In the Big Data era, is it all about having tons of data? or is it about having good aggregation tools?
Tons of data is useless without good aggregation tools, so both are important.
:: «Crowdsourcing Economy»: a bubble or the future? Any particular field where you are more optimistic? Healthcare (CrowdMed), credit ratings (Cdling), market research (Infosurv, Indingo), corporates (Consensus Point, QMarkets) gambling (Predictious, SMarkets), scientfic purposes (PredictWise, SciCast) …
I believe that crowdsourcing is here to stay because it offers so many practical advantages over traditional methods in terms of accuracy, cost, time, scalability, etc. I’m most optimistic about crowdsourcing in healthcare, but perhaps I’m biased… you should crowdsource the answer to that question for bias cancellation 🙂
:: Finally … where would you like to see CrowdMed in 2 years time? Before CrowdMed, a patient with an unsolved medical mystery only had two options: see more doctors or do their own research online. In 2 years time, I’d like to see CrowdMed universally considered as a third option, combining the human expertise, knowledge, and intuition of doctors with the speed, convenience, and accessibility of the Internet.
