Instill Science — Grant Report #1

Project Update

At the beginning of the grant, we had this idea that we could build a better, peer-led Peer Review system. We spoke to scientists who were eager to participate. But once we had manuscripts in hand, it turns out that people love the idea of helping others improve the quality of their science and research output — not the idea of conducting peer review. Even merely mentioning "peer review" elicited a lot of negative reactions!

We've had to change our approach several times, and re-think how we could create a science community that could help itself raise the bar for the scientific and publishing output.

Along the way, we've successfully organized a week-long, hybrid conference for 400+ bacteriophage researchers from around the world, many from countries in the Global South; we've built a rapid web app prototyping system for community building (and week-long, hybrid conferences) on top of Airtable, Notion, Retool, and Supabase; we've interviewed students, post-docs, professors, and journal editors about fixing Peer Reviews; and we've re-thought, pivoted, scrapped, and restarted our concept many, many times.

But now we're finally converging on an idea that seems to work, and we have 146 early adopters to try our latest iteration on! 😬 😅

Progress on objectives

Peer Reviews are a difficult subject

When we first started, we set out to build a community for more constructive, helpful peer reviews, meant for the manuscript authors. Most scientists have a complex relationship with peer reviews. Every scientist acknowledges that it's broken, and most have even thought about ways to "fix" it.

Peer reviews of today are both a feedback practice and a gatekeeping practice. As a feedback practice, they help authors improve their manuscripts and research, by suggesting copy changes, additional experiments, etc. As a gatekeeping practice, they are supposed to help science maintain its rigor and higher standards. In practice however, this amounts to an unpaid obligation that robs researchers of several hours, if not days, of valuable time. This practice also ends up selectively biasing scientific output towards Western labs, and creates large barriers that block out smaller, underserved and/or less-funded labs.

After talking to students, professors, and even science journal editors, we quickly understood that the term "peer review" comes with a lot of baggage. Everyone already has existing notions of what peer review is and isn't. Any mention of "improving peer reviews" was met with immediate negativity. Everyone balked at the idea of a community that encouraged more peer reviews. So we had to dance around the word.

Everyone we talked to loved the idea of a community that encouraged researchers to give each other manuscript feedback in an effort to improve quality. You just couldn't call it "peer review." We also found that many responded well to the idea of creating a structured, time-limited (30 minutes max) feedback system, where both the author and "Peer Previewers" understand the expectations going in.

We also found that generally, barriers to publishing high quality research exist prior to the point of publication. Usually, the labs that are having trouble publishing are also the labs that have trouble accessing expensive equipment (like sequencers) and data analysis capabilities.

So, after multiple interviews, multiple redesigns, and multiple reconsiderations of the core value proposition of Instill, we realized we needed to be a more generalized resource for scientists of all stages. We also realized that "underserved" researchers could come from unexpected areas, sometimes even from well-resourced labs.

Improving Access, Finding Collaborators, and Removing Barriers, for underserved researchers

Instead of just focusing on improving manuscript output and publication quality, we've zoomed out to look at creating better resources for science communities around the world. From this community peer review workshop series conducted across Africa (https://scholarlykitchen.sspnet.org/2021/08/23/guest-post-best-practices-and-innovative-approaches-to-peer-review-in-africa/), the authors point out the need to "provide opportunities for [members] to engage in debate-style learning, horizontal peer-to-peer mentorship, individual and collective reflection."

Through our conversations with phage researchers and with our partner Crowdfight, we realize that peer-to-peer training, mentorship programs, structured guidance, and efforts to increase representation are all necessary building blocks. Activities like both top-down or peer-led mentorship, peer-led discussions, as well as experimental and data sharing collaborations between scientists will lead to higher quality research output for all groups involved.

As such, we set out to find strategic partners that fit our mission. Through our partnerships, we hope to build a strong advisors and mentorship network.

Developing the right incentives

After scrapping the idea of "peer review" we also needed to rethink incentives. We found that researchers were not primarily motivated by money, but by winning "status games" that prove the credibility and contributions of their careers. These "games" include the publishing game, as measured in H-index, impact factor, and publication records, which all serve as a proxy for someone's research output. They can also involve other types of contributions that can paint a well-rounded career. These include mentorships, invited talks and seminars, teaching, and aspects of community outreach. (More on status games by Eugene Wei: https://www.nfx.com/post/status-games/)

Originally, we started with the idea that high quality, peer-led constructive peer reviews could be paid for, since they would help a lab save a lot of time and grief from the slow and unpredictable peer review process. Though scientists often like to posit the idea of paying for a peer review in conversation and in social media, this squarely did not resonate with anyone we spoke to, given a real scenario where they had the chance to be paid.

A payment structure, or even a tipping or bounty system paints the peer review as a job: as a "paid service to be rendered" rather than a community service chore that every scientist needs to perform "for the greater good." Any sorts of direct payment destroys the illusion that peer reviews are a necessary obligation. We also found that prices are very hard to establish for an audience that exists in virtually every economy around the world... $20 can be overwhelming for someone in Africa but also underwhelming for someone in Switzerland.

We also considered using Coil for performing tasks and "services" like peer reviewing, but because Coil only rewards time spent on page, it doesn't fit as a payment structure.

Instead, we were looking for Reddit, Stack Overflow, and Medium for inspiration from their coin and upvoting mechanisms, as a proxy for how much work an individual is contributing to the platform. Payouts could then be made based on the number of votes, which would incentivize more positive behavior from the members.

We also very recently came across an a16z article on reputation-based rewards systems (https://future.a16z.com/reputation-based-systems) that we could see working with our platform. We could potentially create a payout system based on amount of work performed (for example, number of Peer Previews conducted per month) relative to the entire "pot" of that month. The size of the pot could be relative to the number of participants, or members, of the community.

Implementing these ideas is difficult given the current state of Web Monetization and Coil, but we're very eager to build out some of these ideas in Rafiki.

Establishing a long-term community-led infrastructure

To address our requirements of building a sustainable, affordable, easily operated community specifically for scientists, we looked through the many options that currently exist. We found that existing solutions are either too complicated and expensive to set up (Forem), too opinionated (Discourse), or too closed or enterprise-facing (Circle.so).

We then turned to inspirations like Hacker News, Product Hunt, Stack Overflow and Quora, to identify how they operate at the intersection of "social media" and technical resources and knowledge bases, and created a content and feature inventory.

We had to develop our own architecture, based on off-the shelf tools that have generous freemium offerings, like Airtable and Notion, and using them to develop simple schemas that can easily be ported from Airtable to various other databases like MongoDB, Postgres, or even Google Sheets. We also needed the tool to be natural sciences friendly, which meant that both user and data interfaces need to be user-friendly.

Even though the current prototype stack runs on Airtable and Notion APIs, and is built on top of Retool, the idea is to eventually move to something more robust and free or cheap to host. The idea is to eventually run a Forem-like micro platform for small science communities, but with a fully no-code experience.

Key activities

Evolving Peer Reviews through User Centered Design

When we started, we knew that peer reviews are often a touchy subject, and that we would need to evolve our concept as we spoke to more people. During the design process, we interviewed a variety of members in our Phage Directory community, from PhD students all the way to senior professors and even journal editors.

Through several iterations and conversations, we landed on the idea of a "Peer Preview". The name is chosen to reflect and invoke the idea of structured, faster peer review model, meant to help manuscript authors get a second pair of eyes before submitting to a journal for full Peer Review.

The Peer Preview system starts out as a request from a manuscript author. Members of Instill respond to the Peer Preview request, and if selected, give the author structured feedback through the feedback form. As part of this system, we developed the "SCOR Card" system to help researchers quickly assess papers across significance, clarity, originality, and rigor, based on this paper: https://arxiv.org/pdf/2109.09774.pdf. Requesters are given a dashboard where they can monitor the feedback as it rolls in.

Building a formalized mechanism for finding the right collaborator

We are also developing a Collaborator Search system where we want to make it easier for researchers to find other labs to collaborate with, which includes sharing data, samples, performing new experiments, etc. in order to further develop research and experiments. Through our conversations, we found that many labs from the Global South lack resources and equipment that we take for granted in the West. This could range anywhere from Illumina sequencers to pipette tips, filters, and 24/7 access to electricity. Through a formalized mechanism to find trusted collaborators to send samples, we could help accelerate labs' research progress.

Designing Stack Overflow for natural sciences

Currently we run a Slack group for phage biologists (https://phage.directory/slack), and we monitor and answer several massive WhatsApp groups' phage questions. We have wanted to build a forum system specifically meant for natural sciences, that borrows the best ideas from Stack Overflow, Quora, Discourse, Circle.so, Hacker News, Product Hunt and Reddit. This tool addresses needs similar to those that Stack Overflow are addressing — the "basic" questions that everyone has, but don't know who to ask or who to turn to. This kind of initiative will hopefully help more researchers get past the years-long initial learning curve that most biology PhD students encounter. Furthermore, this platform makes more sense for Coil monetization than the other ones.

Building Communities & Partnerships

To get closer to our target communities, we've partnered with many communities in the phage biology space. We've found groups that align with our mission, and we all share the aim of improving access to science resources. Through the partnerships, we've learned about the challenges their members are facing, which have informed our product designs.

Our partners are:

• Phagebiotics Research Foundation: https://www.phagebiotics.org, which runs the Evergreen Phage Meeting, and trains many undergrad students and researchers from underserved areas of the world.
• PHAGE Journal, by Mary Ann Liebert: https://home.liebertpub.com/publications/phage/652
• Phages for Global Health: https://www.phagesforglobalhealth.org, which trains many phage researchers in Africa and South East Asia
• Crowdfight https://crowdfightcovid19.org, which boasts more than 45,000 community members and volunteers in the fight againt COVID

As we grow and develop our project, we aim to partner with and onboard more communities, such as the Africa Phage Forum, the International Bacteriophage Research Consortium, and others.

Evergreen, an international hybrid conference for phage scientists

Through our community-building efforts, we were approached by the Phagebiotics Research Foundation (https://www.phagebiotics.org) to be the technical partner for the Evergreen Meeting (https://evergreen.phage.directory). Evergreen, one of the major academic organizations in phage research, wanted to run a hybrid conference for phage researchers internationally, which we'd never run (or attended) before.

We joined forces because we both share the aim to provide access to mentorship and resources for researchers from underserved countries (India, SE Asia and African countries). This event introduced us the many new phage communities, as well as challenges that researchers from the Global South face.

For our partnership, we built the web infrastructure for conference that hosted about 500 participants from all over the world... in about two weeks. We devised a ticketing system, payment flow, organized world wide meet and greets, and made many introductions among many phage researchers.

To accomplish this, we devised a system based on Airtable and Notion that let us quickly and collaboratively input the data, like updating the schedule in real-time, adding conference abstracts and 100+ hours of webinars. This system eventually became the backbone of Instill's prototyping system.

Evergreen Github: https://github.com/janzheng/phdir-seminars-evg

Evergreen website (it's still live, and soon we'll make all webinars open access): https://evergreen.phage.directory

Technical Activities

We've built (or heavily modified) a set of tools that powers our rapid prototyping system. We originally looked at pre-built communities like Forem and Circle.so, but deemed them too limited, complicated, or too inflexible.

We therefore decided to roll our own group of tools that let us test our different iterations and value propositions early and quickly. In our setup, we use Airtable as the main document data source, Notion as our site CMS, Retool for rapid UI and workflow development, and Supabase for data like scores, upvotes, and comments. The entire system is deployed on Vercel as a serverless platform, with some code running as APIs on top of Cloudflare Workers. This setup keeps our server and website costs to an absolute minimum. **

The Github repos that power Instill:

• Instill Github (latest version): https://github.com/janzheng/pdn-instill
• Cytosis - an Airtable API wrapper: https://github.com/janzheng/cytosis
• svelte-notion (forked) - A Svelte renderer for Notion blocks: https://gitlab.com/janzheng/svelte-notion.git
• notion-api-worker (forked) - A Cloudflare Worker for fetching and caching Notion blocks: https://github.com/janzheng/notion-api-worker

Communications and marketing

Because peer review and manuscript feedback is a sensitive topic to many researchers, we haven't made a lot of announcements about our product launch. We launched our first version to our newsletter, Capsid & Tail (https://phage.directory/capsid/instill-launch), and tweeted about our product. In total, our early adopter list has 146 members. However, we haven't onboarded all of them yet, as we've been speaking with many of them one-on-one to make sure that our product is appropriately addressing their needs.

Most of our marketing, communications and lead generation are organic. Through our community and our partnership work, we are already connected with thousands of researchers within the phage research field. Our partnership with Crowdfight extends our reach to their network of 45,000+ scientists.

What’s next?

With a better understanding of how researchers prefer to give and receive feedback, we're onboarding users onto both our Peer Preview and Collaborator Search programs.

From the Product-side, we're:

• Building a topics/commenting knowledge base system, along the lines of Stack Overflow and Circle.so, but designed for tight-knit science communities.
• Continuing to evolve our Peer Preview and Collaborator Search concepts with more interviews with researchers and design iterations.
• Designing and building an upvoting, incentive, and rewards system with our researchers. We plan to use both Web Monetization/Coil/Rafiki, in addition to designing ways to show how much impact each community member Is providing the community.

We're also excited to onboard hundreds of researchers from our waiting list, but we're doing this carefully as to not overwhelm our system. Once we feel the product is maturing, we'll be expanding our system into more research areas with the help of Crowdfight.

We will also document more of our learnings and findings, product ideas, and our technical work in our blog at https://instill.xyz/blog

What community support would benefit your project?

The most time consuming areas have been understanding how to implement advanced features of Web Monetization (e.g. direct payments), and other wallet technologies, past Coil features. The other very time-consuming area has been coming up with a simple flow of getting users to adopt a Web Monetized wallet and subscribe to Coil.

We would love if there were some technical communities, working groups, or workshops in the Web Monetization that have successfully built production-ready payment systems other than Coil. I'd also be interested to learn if anyone's thought about creating DAOs with Coil, Web Monetization, and XRP.

Also, if you'd like to collaborate in any way, reach out to us on Twitter (@phagedirectory)!

Additional comments

We're eagerly waiting to adopt a direct payment system (e.g. Rafiki). With a system like this, users could tip each other and create bounties for new work. With direct payments, the platform itself could also set aside a rewards system based on how many points users have accrued within a given time. The payment system could also collect membership dues every month, which would be used to sustain the project and pay back members.

And we'd like to thank all the professors, PhD students, journal editors for getting us to where we are, as well as the Grant for the Web!

Relevant links/resources (optional)

Here's a list of links mentioned throughout the report:

• Our website: https://instill.xyz

The Github repos that powers Instill:

• Instill Github (latest version): https://github.com/janzheng/pdn-instill
• Cytosis - an Airtable API wrapper: https://github.com/janzheng/cytosis
• svelte-notion (forked) - A Svelte renderer for Notion blocks: https://gitlab.com/janzheng/svelte-notion.git
• notion-api-worker (forked) - A Cloudflare Worker for fetching and caching Notion blocks: https://github.com/janzheng/notion-api-worker

A few readings that have taught us a lot:

• Inconsistency in Conference Peer Review: Revisiting the 2014 NeurIPS Experiment. https://arxiv.org/pdf/2109.09774.pdf
• Best Practices and Innovative Approaches to Peer Review in Africa: https://scholarlykitchen.sspnet.org/2021/08/23/guest-post-best-practices-and-innovative-approaches-to-peer-review-in-africa/
• A Novel Framework for Reputation-Based Systems: https://future.a16z.com/reputation-based-systems/
• Status Games: Engineering Scarcity in a World of Abundance: https://www.nfx.com/post/status-games/

Other projects that inspire us:

• Peer Community in: https://peercommunityin.org/
• PREreview: https://prereview.org/
• ASAPbio: asapbio.org