The Interledger Community 🌱: Gaia Dempsey

Metaculus Futures: Building a Sustainable Model for Forecasting — Grant Report #2 (Final)

Gaia Dempsey — Wed, 30 Jun 2021 21:54:03 +0000

Project Update

2021 has been busy for Metaculus so far! As a growing social enterprise, we've been busy building tools that support and enrich our work within two primary fields: we are both a community platform dedicated to providing infrastructure for our active forecasting community and, and a technology + strategy partner to non-profits, government agencies, and institutions tackling high-impact cause areas. Empirically-validated forecasts are at the heart of what we do.

To support both of these organizational roles, ensuring that our platform has a rigorous approach to scoring and incentivizing accurate forecasts is crucial. To this end, we significantly updated our tournament scoring rules earlier this year, aligning tournament prizes with a novel incentive-compatible framework (for forecasting geeks: dig into the blog post for if you're interested in learning more about scoring the accuracy of forecasts, and rewarding skilled forecasters).

Building on the foundation of incentive-compatible scoring rules for tournaments, we set out to create a framework specifically in service to our nonprofit and institutional partners. In April, we launched Forecasting Causes, a mechanism that makes it easier to get cause-related forecasting tournaments onto our platform. It thereby increases the direct connection between our forecasting work and decision-making within organizations and community ecosystems working on crucial cause areas. Feeding two birds with one hand, our cause framework was specifically designed to both increase our overall impact, and improve our ability to detect and measure it.

Utilizing this new framework, we have released three new cause-related forecasting programs — an Alt-Meat forecasting initiative with the nonprofit Good Food Institute, the Keep Virginia Safe Tournament in partnership with the Virginia Department of Health, and the Nuclear Risk Tournament in partnership with nonprofit think tank Rethink Priorities. All three initiatives, which will last either 12 or 24 months, have received robust engagement and participation from the forecasting community and will be expanded later this year.

In addition to all of this cause-related work, we're also exploring business opportunities that will utilize the same core infrastructure, tools, and intelligence we are (co)-creating on the platform. Specifically, we are experimenting with aggregate crowd forecasting as a means to build a trade signal, via the newly launched Trade Signal Tournament. This project has a really unusual and fun twist: an elected Community Trader who will use the forecast data generated by tournament participants to grow the overall prize pool, and thus the available rewards for the best forecasters. Much meta. Will it work? We don't know! We'll be able to share results in three months, but until then, I can say that we definitely expect to learn a great deal about the potential efficacy and value of Metaculus forecasts to inform market investment strategies.

In addition to this exploration of the finance space, we've also been quietly nurturing a couple of small pilot programs in education, and are excited to deepen our connection to these projects.

Finally, in broad support of all our recent efforts, for much of the last couple of months we’ve been working on a major site redesign. The main goals of the redesign are:

Mobile-Friendliness: First, we really wanted to make Metaculus a mobile-friendly place. As I’ve noted elsewhere, over 50% of our traffic now comes from mobile devices, so this has become a very high priority!
Enabling New Features: Second, we wanted to develop a scalable, streamlined design language that can grow with us as we add new content types and features to the site. The primary example here is a feature called Metaculus Notebooks, which will be utilized for Fortified Essays within tournaments. We hope that Notebooks and Fortified Essays will enable a direct connection between quantitative and qualitative information on the platform. Unifying these two modes will provide valuable context to forecasters, researchers, and decision-makers (we wrote more about how this will work in tournaments here).
Improving Usability: Last but definitely not least, as we work more closely with partners, we aim to build tools that not only enable science — i.e., forecasting science — but that will also enable great science communication. Ideally, forecasting tools should feel easy to use and understand!

Progress on objectives

While we have pivoted a bit in our final implementation of Web Monetization, overall we've hewn quite closely to the four aims in our original proposal, which were:

The implementation of a Web Monetization integration with the Metaculus platform.
Improving the usability and discoverability of the Metaculus platform.
The development of a communications infrastructure for Metaculus content, including web-monetizable content.
The development of partnerships with organizations that can use Metaculus forecasts in the real world, including in public health and public policy.

We have made significant progress across all four of these aims. The biggest difference in our final project compared to what we had planned is that the type of integration we implemented for aim #1 ended up taking a different form than we'd anticipated.

To expand on that a bit, we had originally planned to utilize the Web Monetization standard to enable an ambitious change to our platform's core functionality: namely, a forecasting bounty system that would enable people to place bounties on individual questions, with various optional parameters. However, once we began the implementation process, we learned that while the WM standard enables streaming payments suitable for content consumption, it did not support the sort of directed micropayments that we would need in order to adopt the framework as part of a functioning bounty system.

While we would have loved to build the system we had originally imagined, our new solution connects the Web Monetization ecosystem to our new Forecasting Causes framework and thereby builds support for one of our core programs, as well as creating new awareness within the Web Monetization community of novel ways of supporting altruistic, high-impact projects.

Key activities

Specifically, we have added Coil pointers to all our Forecasting Cause-related pages, in order to direct Coil payments to Cause-related prize pools. In this way, we are facilitating crowd support for a community of forecasters who are contributing to and generating an empirically validated shared pool of knowledge in high-impact cause areas!

Communications and marketing

You'll find write-ups on all our most important work and updates on our company blog, and we have a fairly active Twitter presence. We send out regular newsletters to over 17,000 subscribers, which you can join by signing up for a Metaculus account or simply subscribing using the field in the footer.

What’s next?

What's next on our to-do list? Here's a high-level overview:

PRODUCT

Launch our major redesign in a few weeks time.
Continue to run pilot tests in the education space and better understand what bringing a forecasting product to market in the edtech ecosystem would entail. As a next step, we'll be supporting a semester-long course in forecasting with the curriculum specifically developed to utilize Metaculus software.
Analyze the interim and final results of the Trade Signal Tournament over the next 2.5 months, and evaluate whether investing in a crowd-based trade signal utility would be truly worthwhile.

IMPACT

Develop a rigorous impact assessment framework that enables us to model and measure our impact in the cause areas we work in.
Develop a standardized operations process for supporting researchers, nonprofits, and other institutions in their social impact efforts, via our technology tools and strategic foresight frameworks.

What community support would benefit your project?

There are so many ways that interested community members can get involved!

My first suggestion for those who are excited about learning and practicing forecasting would be to sign up for a forecasting tournament on a cause you care about. Participating in a tournament means spending some time thinking about the body of questions, doing research online to inform your forecasts, and ultimately submitting your best estimates in response to each question. You can check out our fun tutorials here, find forecasting resources and best practices here, and our Community Guidelines for respectful and positive interactions with other forecasters and Metaculus community moderators on the platform here.

You may still be wondering, what does forecasting in a tournament actually look like in practice? As an example, Round 1 of our Nuclear Risks Tournament currently has 27 open questions. One question asks how many countries will expand their nuclear arsenals by at least 10% by 2024, and provides an estimate of the nuclear warheads that each country with nuclear capability possesses, according to the Federation of American Scientists.

An information source provided by a forecaster and linked to on the question page provides a useful starting point for research: the Washington Post has reported analyst estimates that China is building more than 100 new missile silos in its western desert. Given that the Federation of American Scientists – which is also the information source that will be used to resolve this question, currently estimates China's arsenal to be 350 nuclear warheads, an increase of 100 silos seems like it has the potential to represent an increase of more than 10%, thus crossing the threshold for this question. But how many nuclear warheads will actually be housed within the 100 silos? A closer examination of the article reveals that a US military analyst believes that there is "'a very good chance that China is planning a shell game' in which it hides a relatively small number of warheads across a network of silos." Would such a shell game potentially include as many as 35 nuclear warheads? What other evidence can you find in support of, or against, this hypothesis?

Gaining experience and skill as a forecaster takes time, but it can be a very rewarding way to learn about a new topic and develop your reasoning skills. It's also a remarkably general skill that can be applied to almost any of your life. We recommend the following as best practices on Metaculus:

Aim to develop well-calibrated judgment by practicing on forecasting on questions such as the one above
Share your reasoning with other members of the community
Learn through the direct feedback you will receive via our standardized forecast evaluation process (e.g. scoring rules and track records)
If you enjoy the process, come back often! We will welcome you as a contributing member of a global reasoning community that generates valuable information through a continuously improving forecasting process

It may be especially interesting for new forecasters to know that the continuous improvement part of this formulation comes from two main sources: 1) forecasting itself is a learnable skill that people who practice can get better at over time, and 2) the forecasting track records that Metaculus stores for each forecaster are used to fine tune an aggregation algorithm called the Metaculus Prediction, which differentially re-calibrates and re-weights individual forecaster input based on their past performance. So, Metaculus itself is also an ongoing, collaborative data science project that's designed to improve its overall collective accuracy over time.

Community members can also help in other ways:

Become a Forecasting Cause supporter
Give us feedback on our new site design!
If you work in a nonprofit or research institution working on a high-impact cause area, get in touch with us about a potential partnership
If you work in education, we would love to interview you and get your opinions about a new package of forecasting tools that we're considering developing for educators, from high school to grad school
If you're a data scientist or developer interested in helping us further our work, we're always interested in meeting passionate Bayesians, discussing ideas, and exploring new opportunities on our team!

To get in touch with us about any of the above, feel free to leave a comment on the site or drop us a note using our feedback form. Thanks for reading!

Metaculus Futures: Building a Sustainable Model for Forecasting — Grant Report #1

Gaia Dempsey — Tue, 09 Mar 2021 03:01:07 +0000

Image (optional)

Project Update

Metaculus is a science-inspired forecasting technology platform that enables our community to collectively reason better and empowers public, nonprofit, and private-sector entities to predict, plan for and better respond to a range of developments. Our most popular forecasting work covers Covid-19, science and technology progress, artificial intelligence development, the global economy, and much more.

In highly uncertain times, tools such as ours are especially valued: in 2020, a global pandemic, a historically unprecedented US Presidential election, a vacant Supreme Court seat, massive wild fires and widespread riots across the United States, and the most impressive worldwide scientific collaboration effort in history all drove high levels of traffic and participation to our platform. Through moderated community questions and statistically aggregated forecasts, Metaculus enables anyone with an internet connection to access the best current human knowledge about the future.

Our forecasting scoring algorithms keep track of the individual forecasts made by participants over time, and when we empirically identify forecasters that tend to be both accurate and well-calibrated, we differentially weight their input in our optimized community aggregation, the Metaculus Prediction, so that we can single out the signal from the noise with the greatest possible accuracy.

This solution means that brand new forecasters who are learning about judgmental, model-based, and probabilistic forecasting can test and hone their skills on Metaculus without worrying that they are "ruining" the accuracy of the Metaculus Prediction. Our system elegantly provides a training ground for newcomers, a sifting mechanism that identifies the most skilled forecasters, and accountability and transparency for all.

At the beginning of September 2020, we had approximately 12,000 registered forecasters on the Metaculus platform. As of the beginning of March 2021, we now have well over 16,000, meaning that our userbase grew by a full third in the last 6 months.

In that time, we've launched the largest AI forecasting tournament in the world, (the launch was covered by Forbes), partnered with The Economist on forecasting 25 key events in 2021, published an op-ed in The Hill highlighting our forecasting work on Covid variants and our related public health policy recommendations (which the Biden administration later adopted, though probably not just at our behest), and trialled scoring a public figure – Vox co-founder Matt Yglesias – on his forecasts for the year ahead. (This last foreshadows some future planned features related to public figures yet to come.)

Our forecasts are featured in three new regular publications read by forecasters in both business and academia.

At the same time, we've made significant progress on upgrading our platform infrastructure, usability, UX, and ease-of-use.

Since the beginning of this year, we've brought our founding CTO, Max, back on board full-time. His presence has not only resulted in much faster engineering progress overall, but also in scalability being at the core of all our major engineering decisions, giving us all confidence in the long-term viability of our platform tools and capabilities.

Here is an overview of the most important recent feature, design, and overall product updates:

Design upgrades including "Bright Mode" – a toggle-able view that provides a white background and dark text, as opposed to the reverse, which makes Metaculus much easier to read for non-programmers – plus a new clean header design that aligns with modern web application design expectations
Comment Up/Down votes
Moderator Workflow Improvements, including the ability to sort the Pending Queue by questions that individual moderators are actively moderating, by questions that have had no moderation yet, and by newest/oldest. Also, the ability to temporarily suspend user accounts, update question categories, and more.
New Community Guidelines with information about what we expect from platform participants in terms of Etiquette, and what they can expect in terms of Moderation Rules and Sanctions. This represents an important milestone in setting expectations with our users, especially newcomers, as much of the etiquette of Metaculus has developed over time and has been implicitly understood by long-time veterans as a sort of tribal knowledge. Writing it down helps to clarify the behavior we expect to see – treating others with respect, for instance – and levels the playing field for all. As we said in the Guidelines themselves, "We greatly value the contributions of our diverse community of forecasters, question authors, and forum participants, and we hope that these guidelines will promote, enhance, and safeguard a vibrant community forecasting space for many years to come."
Infrastructure upgrades: upgrades to Python 9.6 and the latest version of PostgreSQL, plus MVP automated testing infrastructure and feature flags for beta testers
Possibly the most significant change is a "seismic shift" in our backend architecture that enables three new data objects called Projects, Organizations, and Notebooks. Only with this new db architecture do we feel we can scalably implement Web Monetization.

Progress on objectives

One of the core goals of the project was to implement the Web Monetization standard in order to enable a fully scalable forecasting bounty system on the Metaculus platform.

When we got into the weeds of Web Monetization technology, we found that it only enables streaming payments suitable for content consumption, and not the sort of directed micropayments that we would need in order to adopt the system as part of a functioning bounty system.

As a Plan B, we came up with a way to meaningfully integrate Web Monetization into the Metaculus platform by rewarding question and notebook authorship with the streamed payments of Coil subscribers. Our Plan B still aligns with our core values, which put our community at the center, along with knowledge and truth-seeking, accountability, transparency, accuracy, and collective intelligence.

We would generally have preferred to have the option to stick to Plan A because it would've enabled a truly remarkable new capability on the platform that elegantly solves a major incentive challenge, but Plan B still creates positive incentives that generate value for multiple stakeholders, so it is something we are happy to pivot toward.

In part because of this pivot, and in part because we haven't been able to staff up as rapidly as we'd hoped, we have requested an extension that will enable us to implement Plan B with excellence.

The core Web Monetization implementation is already completed. What remains to be done is mostly centered about UX and UI (which is also why the "Design" line item in our budget has yet to receive much of a dent).

In addition to the platform upgrades described above, a few more are currently underway:

On-platform notifications, which we expect will increase both ease-of-use and engagement
A new search filter capability, enabling users to more easily find questions and forecasts and navigate the platform
Updated Scoring Rule and Track Record capabilities that enable better incentives and more detailed comparisons of forecaster performance
And most significantly, we have been working on a tremendous new capability entitled Forecasting Causes, which is a bundle of features that allow us to work much more closely and effectively with nonprofit partners.

There are many reasons why we have chosen to focus on Causes first. First, our forecasting community includes a strong contingent of self-identified Effective Altruists who are motivated by a desire to utilize rationality tools, including forecasting, to do the greatest possible good in the world. Second, we have strong interest from nonprofit partners in utilizing Metaculus forecasts in real-world decision-making scenarios. And third, the scalable set of tools that we can build for nonprofits will enable us to learn and test out new capabilities, which we believe will be equally useful with modest modifications to three additional core audiences: educators, businesses, and government agencies.

Key activities

There are four major Activities in our proposal:

The implementation of a Web Monetization integration with the Metaculus platform.
Improving the usability and discoverability of the Metaculus platform.
The development of a communications infrastructure for Metaculus content, including web-monetizable content.
The development of partnerships with organizations that can use Metaculus forecasts in the real world, including in public health and public policy.

Earlier sections in this report have given a status update on Activity #1 (specifically our pivot from Plan A to Plan B, which is well-defined). I've also mentioned numerous improvements we've already made and plan to make soon with respect to Activity #2 – in short, we are in the midst of a usability revolution, thanks in no small part to the tremendous effort and leadership of our design advisor and consultant Steven!

Let's focus for a moment then on Activity #3. First, some context. Metaculus has long been a place where the statistically-minded – quants, physicists, researchers - have been happy making rigorous probabilistic forecasts and obsessing over the details of our (fairly mathematically complex)scoring rules (click on "Here are the details" to get at the underlying equations). The not-so-statistically-minded, on the other hand, have often told me that they find the platform to be a bit "intimidating."

Activity #3 is for this second group of people (and actually, so are Activities #2 and #4).

The goal of our new communications infrastructure is to take a platform that has science, mathematics, and data engineering deeply ingrained in its DNA and produce outputs that are legible and interesting to the average layperson. If this seems difficult to do, consider that it is what science journalists do, what the entire field of science communication is dedicated to.

There are multiple levels at which this goal may be approached. I think of them in terms of a) design and user experience – making the experience feel comprehensible and familiar, b) object-level content – for instance, writing about forecasting in a way that's interesting and relatable, and c) wide and diverse distribution – ensuring that we can and do reach people beyond the already-statistically-minded bubble that Metaculus first appealed to.

An absolutely critical piece of infrastructure that underpins the achievement of all three levels a), b), and c) is a feature we call Notebooks. There's nothing particularly magical or mysterious about Notebooks – they should function in a "comprehensible and familiar" way after all – but they need to enable both our team and our user community to "write about forecasting in a way that's interesting and relatable." Meaning, basically, that instead of simply viewing a forecasting question and related probability distribution on say, cumulative two-dose vaccinations by the end of the month you can read an eloquent essay on the topic of Covid forecasts (this one written by our primary Covid researcher, Juan), which puts the forecasts in their proper context.

Crucially, Metaculus Notebooks should be able to fully integrate Metaculus forecasts natively.

As mentioned above, the backend of Notebooks has been built, and the frontend will be ready and tested on approximately Monday 3/22, in time for a forecasting essay contest we plan to launch shortly thereafter utilizing Notebooks, which will serve to familiarize our community with the feature, incentivize the generation of lots of interesting content (there will be several thousand dollars allocated as prize funds for the contest, plus highly regarded judges), and to work out any technical and UX kinks before turning on the spigot of monetization.

In addition to Notebooks themselves, there is a panoply of smaller features that nonetheless support the broader goal of a highly-usable communications infrastructure, including things like forecast embedding improvements, social sharing preview improvements, a bit of font and styling work, and deeper integration capability with the news ecosystem.

We now turn to Activity #4. Over the past several months, we have been developing partnerships with organizations that we simply could not be more excited to work with. I already mentioned The Economist, and though he's not "an organization," I'll reiterate that we're very excited about our recent collaboration with the author Matt Yglesias.

We've also been very effectively collaborating with Lehigh University's Computational Uncertainty Lab on Consensus Forecasting to Improve Public Health: Mapping the Evolution of COVID-19 in the U.S. – more on some of the latest results of this collaboration can be read in this recent LU write-up.

Several other important partnerships are well underway and will be announced soon, but since this is a public forum, I won't mention them here just yet. I would be happy to discuss them privately with the GftW grant administration team.

Communications and marketing

While we frequently discuss our work in public, I'll focus this section just on the marketing budget envisioned for this project.

Once the Notebook capability is up and running and Forecasting Causes have been launched, we are planning a podcast ad campaign. (We plan to test out podcast ads since we've found declining returns over the last few months with Reddit, Facebook, and Google ads.)

We will also likely need to come up with a viable marketing alternative to the account drops we had envisioned utilizing as a promotional tool in web monetization Plan A which involved a scalable bounty system, and identify a more suitable promotional tool that aligns with the structure in our Plan B, namely a reward system for the top monthly authors on the Metaculus platform. We haven't nailed down exactly what this should look like yet, but perhaps an expanded essay contest would work well here.

What’s next?

We plan to complete and launch Notebooks, complete the implementation of our web monetization Plan B, which mostly means the completion of the relevant UI/UX work and executing the marketing launch (unless by some miracle we find out that there's a way for us to circle back to our Plan A, perhaps based on new changes in the WM ecosystem that we're currently unaware of).

We will also launch our first essay contest, as well as the Forecasting Causes capability, as well as announcing our first major FC nonprofit partnership at launch.

What community support would benefit your project?

For technical members of the WM community, we'd love to hear of any updates to the Web Monetization ecosystem that would enable us to build a scalable, micropayments-based bounty system for for forecasting on Metaculus.

If you work with or for a nonprofit and you're interested in probabilistic forecasting and related tools to support complex real-world decision-making, we'd love to hear from you.

Looking further ahead, we're also very interested in working with educators who want to explore the use of forecasting tools in the classroom, at graduate, undergraduate and high school levels. Current experiments underway are in math and physics courses, biostatistics, psychology and decision science, as well as in business schools. We'd be thrilled to work with experts in the biosciences, biosecurity, sociology, complexity science, or anyone interested in multidisciplinary academic efforts.

And of course, we always love UX feedback from new users of Metaculus – fresh eyes are always incredibly helpful!

Additional comments

Thanks for reading our report!

Relevant links/resources (optional)

Metaculus
Pandemic Metaculus
Economist 2021 Series
Matt Yglesias collaboration

Hello from Metaculus!

Gaia Dempsey — Fri, 18 Dec 2020 19:21:56 +0000

Hello Web Monetization community! 👋

We've been busy the last couple of months setting up our basic web monetization integration, building new features that will be web monetizable, and making general usability improvements.

We've also launched our biggest forecasting tournament ever – Forecasting AI Progress, which has already attracted more engagement and new users into the Metaculus ecosystem. There's $50K in dedicated prize funds for this competition, so if you are interested in AI and want to get into forecasting, definitely check it out!

One of the biggest new features we've developed is the Metaculus Notebook, a place where our users will be able to write analysis about existing forecasts, their rationales for new forecasts they're developing, or other content of interest to our community. And we're very excited to reward authors of Metaculus Notebooks with streaming web monetization payments!

In addition to this WM opportunity, we'll also enable payments to our top 5 most valuable Question Authors, as judged via the activity and engagement of Coil subscribers.

While we originally intended to utilize micropayments as the basis for a bounty system, our initial experiments have led us to the conclusion that the technology isn't quite ready for this type of implementation, so we've come up with an alternative plan.

If you’re interested in getting an overview of what our integration plan looks like, how we’ll reward Metaculus Notebook and Question Authors, and what we’re excited about for the future of web monetization, please check out our lead developer, Matej’s presentation on the subject. We’re gearing up announce, launch, and promote our user-visible Web Monetization features in the new year.

Needless to say, if you have thoughts/feedback to share about our plans, or just want to geek out on Web Monetization or probabilistic forecasting, please don't hesitate to reach out to our team!