Pioneer, evangelist and solid defensor of prediction markets. He is the benchmark for any topic regarding prediction markets, and he is the one able to discuss even with Nassim Taleb explaining which are the weaknesses, and which are not of this technology. Robin is the right choice to explain the potential benefits for the society of using prediction markets.
I was honoured to do a couple of interviews on Skype with Robin (yep! I said two, as I’m not a techie and my first try to record it wasn’t successful). We talked about the inception of prediction markets, his own expirience using them in coporations, and of course, about the potential future of this technology.
There is no doubt that Robin is one the most influential ecnomist nowadays. He shares recognition with Tyler Coewn when anybody want a delibarated thought on how the future may be in our economy, Robin is the man to ask. You will find him participating in everything related to forecasting based on the wisdom of crowds. His prejects have been funded by the government (a couple of his «markets» have been funded by IARPA, both PAM late in the 90’s and SciCast these days), or they’ve developed into a private company (he is Chief Economist of Consensus Point).
By the way, did I tell you that Robin gave a speech on «Prediction Engines» during the last Microsoft Summit 2013, talking just after the inaugural conference of Bill Gates. Actually, Microsoft Research is one of the very first entities to invest in this new technology (we should mention here David Pennock and David Rotschild as key economists too), together with companies like Google, Yahoo!, Intel or HP. No doubt we are talking about a promosing new way to forecast any kind of event. We will explote its stregnths and weaknesses.
If you fully go through this in-depth interview, you’ll be completing succesfully a serious introduction to this field. Next thing you may do is just start predicting the future, today! Join Scicast.org and learn what will be new trends in sciencie and techonology.
– Should we say «wisdom of crowds» or «wisdom of markets»? Where is it the predictive power, if any?
Prediction markets are a mechanism, an institution, to be used by a pool of participants and a set of questions. And, in order to get this pool of participants to answer you need to give them incentives. Holding constant the participants, the incentives and the questions, we can vary the institution. We can imagine different sets of institutions in that place. It could be a political committee, a voting pool, or a prediction market. So the claim about prediction markets is that, holding constant these other things, the prediction market would be a better institution, giving you more accurate answers to your questions.
Prediction markets are not about a particular kinds of persons or questions, or even a particular kind of incentives: they are about how to combine those things to get good answers.
When people talk about the wisdom of crowds, they say that ordinary people know more than what they are being given credit for, or that there are some self appointed experts who aren’t really the best experts, and that it would be good for society to listen more to the crowd of ordinary people than to the experts. There is maybe some truth to that, but prediction markets aren’t about that question, because prediction markets can be used by a small number of people who know a lot as well as by a large number of people who know a little.
More fundamentally, prediction markets are a way the community selects and decides who are the real experts. So with a prediction market you are less sensitive to who you include. If you include a bunch of ordinary people and a group of experts, they will choose how much to participate based on their own expectation of how well they will do. On a topic where real experts have deep knowledge, ordinary people will back off, letting them answer because they really know more. There will also be topics where experts realize they cannot know anything better than anybody else, and when it comes down to betting, they will back off, shut up, and find excuses to do something else. In that case, the ordinary crowd will speak up, because they are the relevant experts.
– Do prediction markets collide with the (in)efficiency of financial markets?
Prediction markets are much more specific than markets in general. This is not about all potential markets, it’s about speculative markets. And speculative for a particular purpose. Most financial markets function to trade or hedge risks, and they have a side effect of aggregating information, but they are not purely created for that. And that’s not the main reason why their customers and participants are there.
So prediction markets are speculative and their purpose is aggregating information. There is some set of customers who want to know something, and are willing to pay for it. And that’s what creates a prediction market. So it’s a similar mechanism with a different purpose.
Nevertheless, the experience of financial markets says that sometimes they are inaccurate. It’s clearly true that speculative markets are not always accurate about everything. But the right comparison is about speculative markets in reality compared to another institution in reality. There is a large literature in financial markets that tries to compare real financial markets with ideal financial markets, or a theoretical idea about them that doesn’t make mistakes nearly as often. But the practical question is: which other actual practical institutions can we use instead? And when we compare the performance of speculative markets with that of polls and committees, the speculative market approach seems to do very well in comparison.
– Does money really matter in Prediction Markets? Justin Wolfers says no.
Everyone would agree that you need incentives that the participants care about, and money is often an effective way to motivate people to do things. When people are not motivated at all, they don’t do much of anything, and in prediction markets you need people to be motivated to join and to constantly pay attention. They need to care about winning or losing, so there needs to be something at stake. It could be reputation among a group they care about, it could be a status, it could be t-shirts, meeting with the CEO or extra free time. There are lots of things people could care about, but you need something that motivates all the group.
But if you include a lot of people in the group that do not know each other, it’s hard to find something they care about together besides money. So money is a very flexible and convenient incentive to get people motivated, but it’s not the only thing they care about.
– Do we need «another» Intrade?
I would really approve of having more public prediction markets with real money, but there are still a lot of regulatory barriers. Intrade was a prediction market in the sense that it aggregated information for the topics they were offering, but it not was not a prediction market in the sense that it did not offer to pay people primarily to answer those questions. It was more like a traditional betting market, stock market or other financial market, in that prices were side effect of people trading for other reasons. So I’d say it’d be good to have more financial markets in general on different topics, but that’s not the same as having more prediction markets. In order to have more prediction markets we need markets where there is a particular customer who wants the answer to a question and is willing to pay for that, and that entices the other participants of the market.
– So, would you say that there are different kinds of prediction markets?
Yes. There are two basic kinds of prediction markets. The first one are the markets that are open to the public, where everybody in the world can contribute to a question. The second kind exists within an organization, a corporate or a firm. This second type of market is focused on the topics that are of interest of the specific organization, and the often function just with inner participants from the organization.
These regulatory laws refer to the first kind of prediction market, but the second kind of prediction markets are actually legal. So many firms have and continue to develop prediction markets within themselves, and those markets are the ones I have the most hope for demonstrating their predictive power; once we demonstrate this, we could move towards more public markets.
– Why do we have prejudices against bets? Is the world nowadays more prepared to understand the information aggregation power of betting?
We’ve known about bets and gambling for many centuries. For a very long time, gambling was a common activity among many people, something that could be done for entertainment. But since it was not considered very productive, laws and rules began to be put in place preventing people from gambling, so that people wouldn’t lose time and resources on it.
Most of the financial markets we have today were at one time prohibited, also accused of being unproductive gambling. So stock markets, insurance markets, commodity futures markets and options were once illegal and their advocates had to show that, even though they might look like gambling, they had a useful purpose and they deserved a separate legal protection and a regulatory coverage.
And this is the hope of prediction markets: to carve space in people’s mind for not being considered useless gambling, since they are useful in a new way and have a purpose.
– You have mentioned that trying to sell internal prediction markets is tough. Which are the barriers of entry you may have noticed with your company Consensus Point?
Prediction markets inside firms have consistently given more accurate forecasts than the previous forecasting systems that the companies had in place before their arrival. They have a very consistent accuracy trackrecord. Lots of major organization make a lot of decisions that depend on forecasts, so forecast accuracy is an important contribution to be efficient in those decisions. In that sense, firm-specific predictions markets should be very useful in a wide range of purposes.
However, starting prediction markets inside companies is not easy. First, it’s difficult to convince organizations that prediction markets can improve their forecasts. Then, when companies decide that they are interested, they usually do so because they want to do something new and cool, something sexy so as to come up in the news. But when we go and start to set up the market and try to put the questions that are central to the organization, they usually shy away. They say things like “That looks a little too sensitive and political”, or “We don’t want predictions markets on questions that are that sensitive, at least at first”; “We want questions on something else that people would feel more comfortable with”, and they choose questions that are far away from their key activities. And it sometimes happens that when they get the right answers from the market they realize that they didn’t care that much about the things they asked, lose interest and go away.
Other times, they realize they do care about them but they care too much. Some of the most dramatic successes in prediction markets within organizations are about deadlines and project completion dates. And very often prediction markets have very accurately predicted that a deadline would not be matched even though everybody was internally saying the contrary. And, paradoxically, those successes do not encourage corporations to use prediction markets more. When this happens, someone has been embarrassed, and it’s usually the person in charge of the project to miss the deadline. After the failure, the responsible person usually tries to argue that “no one could have known or foreseen that”, that “it came out of the blue and therefore I shouldn’t be blame for the failure”. That’s a hard story to tell when there is a track record warning “this is going to fail, this is going to fail, this is going to fail”. As one would expect, these people don’t want prediction markets to be in place, and are usually capable of convincing their manager to get rid of prediction markets so that they not embarrassed again in the future.
– Tetlock and Gardner tell us a similar story: “people have aversion to acknowledging their ignorance”. Do you agree with this statement?
Organizations seem surprisingly disinterested in the real accuracy of their own predictions, and there are a lot of explanations we can offer.
One possible reason is jsut that organizations don’t want to admit that they don’t know, and that’s also one of the reasons why randomized trials are not well seen. When you do a randomized trial your must admit that you do not know the answer, and some organizations cannot just admit this.
Moreover, when you want to be able to say “no one could have known what happened”, you have a problem if you have a prediction market where other peopke did indeed know.
There are also other explanations: managers in charge of a project are trying to obtain the maximum effort out of it, and one way to do it is to put people right on its edge. “Only if we all work hard and together we may get to this deadline”. And to pretend you are really on that edge line, the probability of making the deadline should be close to 50%, but if the estimated probability from the market was in 90% or 10%, the workers would be less motivated to work hard in order to make it. So, in order to get the maximum out of employees, the managers sometimes have to mislead them, telling them is more likely they are in the 50% range. When a prediction market contradicts them, that can be an obstacle for the manager trying to get the best effort from their team.
– Tell us a little bit about the project Policy Analysis Markets (PAM), in which you also suffered some obstacles…
In 2003, I was part of a research project that was going to go live with a prediction market about geopolitical events in the Middle East. Two senators held a press conference where they declared to everybody that the government was trying to set up a prediction market about a terrorist attack and that this was a terribly immoral thing that we should stop immediately. In fact, it was stopped within a day without the people who stopped it talking to us about the project, maybe because it seemed to be just an obviously immoral thing to do.
Thus, the problem was not a concern about the information, but about violating a moral principal that I’d phrase like “no one of us should intend to benefit when one of them hurt some of us”. The truth was that this was not a market on terrorist attacks, which was an incorrect accusation, but the politicians decided nevertheless they would get bad press if they waited too long to take the decision.
– What about life insurance policies? We are betting about death and life expectancy. Why don’t we have these moral barriers here?
We had a similar barrier before the existence of this insurance market, but the life insurance industry managed to portray itself as something different and useful: “You are a responsible father who cares about your family, and you will ensure they can continue comfortable after your death; therefore, you should let them bet on your death”. And this is what life insurance is: letting your family betting on your death. Because it seems to serve this important function, it seems to be acceptable.
Once prediction markets start to be more a common practice and people see the positive information benefits that they produce we could reinforce the fact that this is not a generic bet, but we are not yet at the moment.
– People may have difficulties relying on an ordinary crowd when looking for wisdon. What do we need in a crowd in order to expect a good forecast from them?
When you have a question, somebody among the participant should know something about it. That is the key constraint. If nobody knows anything, you’re in trouble. Prediction markets cannot include information that none of your participants know. However, if only a few of your participants know something and the rest of them don’t, that could work out fine, as long as the other people who don’t know much are aware that they don’t know. So, more than the average quality of your audience, the importance lies in the extreme qualities. You should be much less concerned about excluding people that may do harm than about including people who might help. Prediction markets are more robust to “bozos”. If you are holding a political committee with 12 people and 7 of them are bozos, your committee is gonna do badly. In markets, the bozos who know they are a bozo, shut up, and let the other people speak, In a prediction market you can have 90% bozos and it can still work fine because people who do know can dominate the prediction market more effectively.
– This brings us again to Tetlock and Gardner. Is “being a fox” a dominant strategy over “being a hedgehog”? Should we aim to be foxes, learning a little bit of a broad range of things, or should we focus on mastering a single field? Tetlock says foxes predict better, but Adam Smith says specialization is the rule to follow.
If you create a prediction market about questions you care about and you give sufficient incentives to the participants, you just need to get as many participants as you can of whatever sort you can. It shouldn’t be up to you to tell them how to invest, or when to invest or where to invest, or which strategies they should you use. You might give some advice, but you don’t need to be very insistent about it. The whole point of a prediction market is that they will self-select. Strategies that do not work will be punished, and those who do work well will be rewarded; this second group of people will dominate and the rest of people may come to copy their strategy.
So, if the hedgehog strategy is worse, as it seems to be from Tetlock’s analysis, which I perfectly believe, then prediction markets participants will tend not to use the hedgehogs strategy, because they would lose that way. But the big advantage of the prediction market is that you don’t have to decide which strategy they use, you don’t have to decide whether you need more PhDs or or psychology graduates, or if you need only adults or only men. You don’t have to decide what is an and expert: they self-select.
– Forecasts of prediction markets are model-free. We do not impose a model to estimate the outcomes. Are they reliable?
The most straight forward answer is to take a look at their track record and their accuracy, especially if you can compare it to the track record of some other method. If you want to convince people that prediction markets are reliable that track record is by far the strongest evidence you can use.
So there is an interesting question, why are people still are reluctant to use the prediction markets after seeing their succesful track record? One of the reasons is the complex use of credentialism or treatment of experts. On most organizations, we use degrees and positions as a way to decide who is important and who gets to speak, and prediction markets are violating that by speaking without degrees or positions of authority.
In many organizations, people pretend to be trying to collect information in order to make the best decisions overall for the organization. But what they are actually doing is a lot of politics deciding who is in which side and who has the majority of influence, so there is not an actual passion for information within the organization. That’s why there are not that much interested on new real information.
– Tell us something about the very new project SciCast. How is it going so far?
I’m part of a project named SciCast, funded by IARPA, which is an intelligence research part of the US defense agency. You can see how it works on scicast.org. It has hundreds of participants and hundreds of questions about science and technology and it’s a proof of concept of a new technology for doing prediction markets which is called combinatorial prediction markets.
The key difference is that here there is not just a hundreds of questions on which you can express your opinion on, and disagree with other people and change the price.
This means the system gives not just marginal probabilities on each particular answer, the market gives a joint probability distribution on all combinations of answers, which has much more information and much more detail about the relationships between answers. For example, think of this question: “If technology A is achieved, what is the probability of other technologies being achieved as a result?”. This way, we can go and browse those dependencies and get what the market thinks of how technologies depend on each other, after which you can go contribute your opinion about those dependancies.
– Automated market-making algorithms or pure inventory-based marke-making? Which method is better for information aggregation in markets?
A human market maker can be inventory based, or a human market market could base his trades on something else, similarly for automated market makers. Being inventory-based is just a robust way to keep yourself from been cheated in the worst case. So an inventory-based market maker’s main advantage is that they only lose so much, and you can prove that the organizer can limit its losses. So if you expect your market maker to be really stupid compared to the traders that he’s playing against, you can expect they’ll try to outsmart him and take advantage of him whenever they can find it. If you expect to be taken advantage in that way you want to be simple and robust. So an inventory-based market maker is a simple way to avoid losing.
But if you know that you have traders that try to be clever and take advantage of you in any possible way, you can create a market maker that tries to be more ambitious, tries to do more things and might not be taken advantage, you might instead help the market and allow more trades between other parties and provide more liquidity.
– «Crowdsourcing Economy»: a bubble or the future? Any particular field where you are more optimistic? Healthcare (CrowdMed), credit ratings (Cdling), market research (Infosurv, Indingo), corporates (Consensus Point, QMarkets) gambling (Predictious, SMarkets), scientfic purposes (PredictWise, SciCast) …
I’m not very optimistic about the crowdsourcing economy. I dislike the connection made in people’s mind between crowdsourcing and prediction markets. They are separate approaches for separate problems. Prediction markets are a way to buy information if you are willing to pay for it and there area people out there who might have access to the information you want. Crowdsourcing, in contrast, is the idea that instead of paying employees to do things for you, you might get people on the internet to do those things for free if only you manage to make the job fun enough.
Some people have managed to make jobs fun enough to get random people to do it for free rather than having to pay employees to do it. I’m not very optimistic about that as a general way of where the economy is going to go. Most things that need to be done in the world simply cannot be made fun enough so as to be able to fire employees and get other people to do it for free. And these include most of the things you may want prediction markets for.
Prediction markets are a new and powerful technology for paying less to get more accurate information from people, but they do require people get paid in some way.
The long term path is having some big organizations using prediction markets consistently and getting added value. That way, neighboring organizations would start to pick up in similar industries and similar circumstances and the mechanism would slowly spread until lots of organizations used it and wouldn’t think about not using it.
So, generally, my hope is that it becomes like cost accounting is today. Imagine a world where nobody did cost accounting and you proposed that your company do cost accounting. That would look bad. You’d be basically accusing somebody around that he is stealing. And now imagine a world like ours where everybody does cost accounting, and you propose that your company do not do cost accounting. You’d be basically saying, “I wanna steal, so please, look the other way”. That would also look bad.
Similarly we can hope that eventually prediction markets will become a standard that everybody uses, and that it would look bad not to use it In a world where nobody is using prediction markets suggesting that we use them is saying that we have a problem with people not telling the truth, even though we should all want the truth. That looks kind of bad. So I have a hope that eventually prediction markets just become the standard thing you do as a responsible manager on a project, just like cost accounting is the standard way a manager makes its finances. It’s not because it’s a Nobel grand thing that excites everybody; it’s because its looks bad not to do it.
Robin Hanson
Associate professor of economics, George Mason University
Chief scientist, Consensus Point.
