Metaculus Futures: Building a Sustainable Model for Forecasting — Grant Report #2 (Final)

Project Update

2021 has been busy for Metaculus so far! As a growing social enterprise, we've been busy building tools that support and enrich our work within two primary fields: we are both a community platform dedicated to providing infrastructure for our active forecasting community and, and a technology + strategy partner to non-profits, government agencies, and institutions tackling high-impact cause areas. Empirically-validated forecasts are at the heart of what we do.

To support both of these organizational roles, ensuring that our platform has a rigorous approach to scoring and incentivizing accurate forecasts is crucial. To this end, we significantly updated our tournament scoring rules earlier this year, aligning tournament prizes with a novel incentive-compatible framework (for forecasting geeks: dig into the blog post for if you're interested in learning more about scoring the accuracy of forecasts, and rewarding skilled forecasters).

Building on the foundation of incentive-compatible scoring rules for tournaments, we set out to create a framework specifically in service to our nonprofit and institutional partners. In April, we launched Forecasting Causes, a mechanism that makes it easier to get cause-related forecasting tournaments onto our platform. It thereby increases the direct connection between our forecasting work and decision-making within organizations and community ecosystems working on crucial cause areas. Feeding two birds with one hand, our cause framework was specifically designed to both increase our overall impact, and improve our ability to detect and measure it.

Utilizing this new framework, we have released three new cause-related forecasting programs — an Alt-Meat forecasting initiative with the nonprofit Good Food Institute, the Keep Virginia Safe Tournament in partnership with the Virginia Department of Health, and the Nuclear Risk Tournament in partnership with nonprofit think tank Rethink Priorities. All three initiatives, which will last either 12 or 24 months, have received robust engagement and participation from the forecasting community and will be expanded later this year.

In addition to all of this cause-related work, we're also exploring business opportunities that will utilize the same core infrastructure, tools, and intelligence we are (co)-creating on the platform. Specifically, we are experimenting with aggregate crowd forecasting as a means to build a trade signal, via the newly launched Trade Signal Tournament. This project has a really unusual and fun twist: an elected Community Trader who will use the forecast data generated by tournament participants to grow the overall prize pool, and thus the available rewards for the best forecasters. Much meta. Will it work? We don't know! We'll be able to share results in three months, but until then, I can say that we definitely expect to learn a great deal about the potential efficacy and value of Metaculus forecasts to inform market investment strategies.

In addition to this exploration of the finance space, we've also been quietly nurturing a couple of small pilot programs in education, and are excited to deepen our connection to these projects.

Finally, in broad support of all our recent efforts, for much of the last couple of months we’ve been working on a major site redesign. The main goals of the redesign are:

1. Mobile-Friendliness: First, we really wanted to make Metaculus a mobile-friendly place. As I’ve noted elsewhere, over 50% of our traffic now comes from mobile devices, so this has become a very high priority!
2. Enabling New Features: Second, we wanted to develop a scalable, streamlined design language that can grow with us as we add new content types and features to the site. The primary example here is a feature called Metaculus Notebooks, which will be utilized for Fortified Essays within tournaments. We hope that Notebooks and Fortified Essays will enable a direct connection between quantitative and qualitative information on the platform. Unifying these two modes will provide valuable context to forecasters, researchers, and decision-makers (we wrote more about how this will work in tournaments here).
3. Improving Usability: Last but definitely not least, as we work more closely with partners, we aim to build tools that not only enable science — i.e., forecasting science — but that will also enable great science communication. Ideally, forecasting tools should feel easy to use and understand!

Progress on objectives

While we have pivoted a bit in our final implementation of Web Monetization, overall we've hewn quite closely to the four aims in our original proposal, which were:

1. The implementation of a Web Monetization integration with the Metaculus platform.
2. Improving the usability and discoverability of the Metaculus platform.
3. The development of a communications infrastructure for Metaculus content, including web-monetizable content.
4. The development of partnerships with organizations that can use Metaculus forecasts in the real world, including in public health and public policy.

We have made significant progress across all four of these aims. The biggest difference in our final project compared to what we had planned is that the type of integration we implemented for aim #1 ended up taking a different form than we'd anticipated.

To expand on that a bit, we had originally planned to utilize the Web Monetization standard to enable an ambitious change to our platform's core functionality: namely, a forecasting bounty system that would enable people to place bounties on individual questions, with various optional parameters. However, once we began the implementation process, we learned that while the WM standard enables streaming payments suitable for content consumption, it did not support the sort of directed micropayments that we would need in order to adopt the framework as part of a functioning bounty system.

While we would have loved to build the system we had originally imagined, our new solution connects the Web Monetization ecosystem to our new Forecasting Causes framework and thereby builds support for one of our core programs, as well as creating new awareness within the Web Monetization community of novel ways of supporting altruistic, high-impact projects.

Key activities

Specifically, we have added Coil pointers to all our Forecasting Cause-related pages, in order to direct Coil payments to Cause-related prize pools. In this way, we are facilitating crowd support for a community of forecasters who are contributing to and generating an empirically validated shared pool of knowledge in high-impact cause areas!

Communications and marketing

You'll find write-ups on all our most important work and updates on our company blog, and we have a fairly active Twitter presence. We send out regular newsletters to over 17,000 subscribers, which you can join by signing up for a Metaculus account or simply subscribing using the field in the footer.

What’s next?

What's next on our to-do list? Here's a high-level overview:

PRODUCT

1. Launch our major redesign in a few weeks time.
2. Continue to run pilot tests in the education space and better understand what bringing a forecasting product to market in the edtech ecosystem would entail. As a next step, we'll be supporting a semester-long course in forecasting with the curriculum specifically developed to utilize Metaculus software.
3. Analyze the interim and final results of the Trade Signal Tournament over the next 2.5 months, and evaluate whether investing in a crowd-based trade signal utility would be truly worthwhile.

IMPACT

1. Develop a rigorous impact assessment framework that enables us to model and measure our impact in the cause areas we work in.
2. Develop a standardized operations process for supporting researchers, nonprofits, and other institutions in their social impact efforts, via our technology tools and strategic foresight frameworks.

What community support would benefit your project?

There are so many ways that interested community members can get involved!

My first suggestion for those who are excited about learning and practicing forecasting would be to sign up for a forecasting tournament on a cause you care about. Participating in a tournament means spending some time thinking about the body of questions, doing research online to inform your forecasts, and ultimately submitting your best estimates in response to each question. You can check out our fun tutorials here, find forecasting resources and best practices here, and our Community Guidelines for respectful and positive interactions with other forecasters and Metaculus community moderators on the platform here.

You may still be wondering, what does forecasting in a tournament actually look like in practice? As an example, Round 1 of our Nuclear Risks Tournament currently has 27 open questions. One question asks how many countries will expand their nuclear arsenals by at least 10% by 2024, and provides an estimate of the nuclear warheads that each country with nuclear capability possesses, according to the Federation of American Scientists.

An information source provided by a forecaster and linked to on the question page provides a useful starting point for research: the Washington Post has reported analyst estimates that China is building more than 100 new missile silos in its western desert. Given that the Federation of American Scientists – which is also the information source that will be used to resolve this question, currently estimates China's arsenal to be 350 nuclear warheads, an increase of 100 silos seems like it has the potential to represent an increase of more than 10%, thus crossing the threshold for this question. But how many nuclear warheads will actually be housed within the 100 silos? A closer examination of the article reveals that a US military analyst believes that there is "'a very good chance that China is planning a shell game' in which it hides a relatively small number of warheads across a network of silos." Would such a shell game potentially include as many as 35 nuclear warheads? What other evidence can you find in support of, or against, this hypothesis?

Gaining experience and skill as a forecaster takes time, but it can be a very rewarding way to learn about a new topic and develop your reasoning skills. It's also a remarkably general skill that can be applied to almost any of your life. We recommend the following as best practices on Metaculus:

• Aim to develop well-calibrated judgment by practicing on forecasting on questions such as the one above
• Share your reasoning with other members of the community
• Learn through the direct feedback you will receive via our standardized forecast evaluation process (e.g. scoring rules and track records)
• If you enjoy the process, come back often! We will welcome you as a contributing member of a global reasoning community that generates valuable information through a continuously improving forecasting process

It may be especially interesting for new forecasters to know that the continuous improvement part of this formulation comes from two main sources: 1) forecasting itself is a learnable skill that people who practice can get better at over time, and 2) the forecasting track records that Metaculus stores for each forecaster are used to fine tune an aggregation algorithm called the Metaculus Prediction, which differentially re-calibrates and re-weights individual forecaster input based on their past performance. So, Metaculus itself is also an ongoing, collaborative data science project that's designed to improve its overall collective accuracy over time.

Community members can also help in other ways:

• Become a Forecasting Cause supporter
• Give us feedback on our new site design!
• If you work in a nonprofit or research institution working on a high-impact cause area, get in touch with us about a potential partnership
• If you work in education, we would love to interview you and get your opinions about a new package of forecasting tools that we're considering developing for educators, from high school to grad school
• If you're a data scientist or developer interested in helping us further our work, we're always interested in meeting passionate Bayesians, discussing ideas, and exploring new opportunities on our team!

To get in touch with us about any of the above, feel free to leave a comment on the site or drop us a note using our feedback form. Thanks for reading!