Peer Prediction Mechanism: Eliciting truthful information without verification

A mechanism designed to elicit truthful responses from respondents when there is no objective truth, or it is too costly to acquire.

Peer Prediction Mechanism: Eliciting truthful information without verification
Photo by Rob Curran / Unsplash

What is the Peer Prediction (PP) mechanism?

The PP mechanism is a mechanism designed to elicit truthful (i.e., honest) responses from a set of respondents when there reports cannot be verified because: 1) there is no objective ground truth available (i.e., inherently subjective circumstances), or 2) it is too costly to acquire the ground truth.

The PP mechanism relies on the correlation of a respondent’s response with the responses of other respondents to determine how to score their response. If their correlation is high (or rather, very predictive of other user’s reports), then the respondent will receive a payout. If the correlation is low, then the respondent will not receive a payout.

A couple of areas where we can see these problems is in online review sites such as Yelp and Amazon where users review and rank products and restaurants based on their experiences. On both Yelp and Amazon, we have many people giving reviews but we do not know the authenticity o these reviews, and more often than not, users are only giving 1’s or 5’s to the product or restaurant they are reviewing.

Now, can we really say how the user experienced the product or restaurant? Not necessarily. This is an inherently subjective area because the only person who can determine a user’s perception of their experience is the user, and to determine if every user was responded honestly, our costs would increase tremendously (we would have to determine if the user is real and how to find them, if they really went to the restaurant or bought the product, and find hard evidence to support their review).

Then back to the ratings, it is highly unlikely that a product or restaurant can only be rated as absolutely terrible with 1 (the lowest) and absolutely great with a 5 (the highest) so frequently. Why are users not giving 2’s, 3’s, and 4's?

Of course, this problem can also be explained by the fact that only those who have a very good or bad experience with a product or restaurant are the most likely to report their experience. However, was that truly their experience with the restaurant or product? Is it not possible that someone rated a product or restaurant a 1 to intentionally drive away business or a 5 to intentionally drive business towards the restaurant or product?

So if we want more accurate ratings from our users, this is where the PP mechanism can come in.

Issues the PP mechanism must address

The PP mechanism not only intends to deal with users who will intentionally lie, it must also deal with users who are uninterested in participating, users who behave randomly, and users who may collude to get a payout either inside or outside of the mechanism.

What kind of contexts affect the PP mechanism?

The PP mechanism needs to be designed to fit the context. Thus, there are many different PP mechanisms depending on the type of context such as the number of tasks and participants, the heterogeneity of tasks and participants, the number and types of signals (e.g., yes/no, 1–5 scale).

What are the applications of the PP mechanism?

Areas where PP mechanism is applicable include without limitation:

  • surveys (i.e., crowdsourcing),
  • peer assessment,
  • product reviews, and
  • determining the offensiveness of an article.

Specific areas we are interested in include blockchain, education, citizen science, and academia.

Blockchain

The PP mechanism can be used to help determine consensus on information fed to data oracles and to shore up token curated registries. Thus, making it easier for DeFi protocols to operate and reduce instances of malfeasance associated with data oracle manipulation.

Education

The PP mechanism can be used for peer assessment in a Massively Open Online Course (MOOC) where students in a class grade each other’s assignments.

Citizen Science

The PP mechanism can be used as an incentivization mechanism in citizen science projects so that citizens can assess and validate each others work (e.g., citizens labeling images and receiving a payout if their labeling has a high correlation with another citizen’s image label).

Academia

The PP mechanism can be used for peer review of academic articles.

Peer Production Communities

The PP mechanism can also be very helpful for peer production communities because peer production communities emphasize peer processes in the production of their outputs.

Help us investigate the PP mechanism

If you enjoyed this article or podcast on the PP mechanism and you want to know more, please see the Supplementary Material section.

Additionally, please consider joining us (as a member or contributor) and investigating the possible applications of the PP mechanism.

Supplementary Material

Readings

  1. Peer Prediction with Heterogeneous Users
  2. Informed Truthfulness in Multi-Task Peer Prediction
  3. Paying for the Truth: The Efficacy of a Peer Prediction Mechanism in the Field
  4. An Information Theoretic Framework For Designing Information Elicitation Mechanisms That Reward Truth-telling
  5. Token-Curated Registry with Citation Graph
  6. Information Diffusion Enhanced by Multi-Task Peer Prediction
  7. Token-Weighted Crowdsourcing
  8. Incentivizing evaluation with peer prediction and limited access to ground truth
  9. A Robust Bayesian Truth Serum for Non-Binary Signals

Originally published on 2020-08-11 on Medium.

Updated 2020–12–25