Storage to the People Final Report

Project Update

We are very pleased to announce the completion of our Grant for the Web project to apply the web monetization framework to the data storage use case.

Our final product for this funded project was a plugin to Etherpad that allows a user to seamlessly save a version of their pad to a participating data storage provider if they have Coil running in their browser. Permanent.org is the initial participating data storage provider for the purpose of this grant.

The system in action can be found at https://pad.opentechstrategies.com. We elected to install the etherpad plugin on a non-Permanent Etherpad server as a way to demonstrate separation between pad and storage provider.

We believe this decentralized approach is an essential feature of our result and this workflow is a demonstration of a more general solution that can be extended to any content-generating application. Our next application of this model will be with the Hyperaud.io project.

The plugin is fully functional for the purpose of closing out this funded project, but there are some important user experience improvements we still need to make in order to make this application user friendly and useful for our members and the general public.

Here are a couple of important notes that will help you test and use this plugin while we continue to iterate towards a truly finished product:

• If you have the Coil extension installed and configured in your browser, you can log in with your Permanent credentials using the Etherpad plugin under the export menu, enable sync with your account, and see etherpad changes show up in your default Permanent Archive under the Apps > Etherpad workspace folder.
• Syncing will time out after an hour and must be manually re-initiated because we still need to improve our token refresh and expiration approaches. You can do that by deleting the “​​permanentToken” cookie from your browser. Once it is deleted, you will be able to login again. this will be addressed in upcoming releases.
• Logging in takes you out of the pad you were just in and you must manually return to that pad yourself so save/bookmark the pad name/url before logging in to Permanent. We will improve this with an auto-redirect in future releases.
• The Permanent login screen is not currently styled yet and appears as the simple, default FusionAuth login UI. Don’t be concerned by that.
• We still have some important cleanup work to do in our Permanent node-sdk and Etherpad plugin repos: working branches will be merged in the coming weeks.
• OTS is planning some maintenance on their Etherpad server. Although some of that can happen without downtime, they will wait a few days after this post to do it.

We will be updating our Etherpad knowledge base article to reflect the current state of the plugin shortly, but the current article serves as a reasonable guide for using the plugin.

Please, take a moment to give our plugin a test for yourself and let us know what you think! Please see the community support section at the end of this post for ways that you can contribute to this work.

Progress on objectives

In this project we set out to achieve the following goals:

1. Design a workflow that builds on the Web Monetization protocol
2. Upgrade the Permanent API to support that workflow
3. Build a simple demonstration application that uses the protocol to store content
4. Document everything
5. Improve understanding of ecosystem gaps

We’re very pleased to report that we largely completed all 5 of these goals during our initial grant period. However, in order to extend the scalability of our solution, we requested a strategic no-cost extension to achieve additional development goals.

In the course of our work we discovered that for the Permanent API to properly support the Coil workflow and integrate with a generic demonstration application on an external server, we would have to make some very big changes to the Permanent authentication framework. So we took a strategic pause on the core objectives in the spring and spent the last six months educating ourselves on how to best tackle this overhaul and executing on it. In effect, we ended up adding a sixth objective:

#6 Upgrade the Permanent authentication framework to to support third-party applications and services.

We can now say that we have completed all six of these objectives.

Key activities

Our team consisted of four key people (moving counter clockwise in the screenshot, starting from the top left):

• Jason Owen, Software Engineering Consultant, Open Tech Strategies – engineering lead on web monetization framework and Etherpad plugin;
• Cecilia Krum, Director of Engineering, Permanent Legacy Fdn – technical direction and interface with Permanent.org API;
• Robert Friedman, Executive Director, Permanent Legacy Fdn – project strategy and grant management;
• James Vasile, Principal, Open Tech Strategies – research lead, rapid prototyper, documentarian;

Some combination of this team met weekly to review goals and priorities, report progress, assign tasks, and make direct progress on grant objectives through the winter and spring of 2021. At the start of the no-cost extension, most of the work became very focused on the implementation of the authentication framework. We joined the Grant for the Web community call in June 2021 to share our work as well as joining a Coil conference. We also presented some of our key findings on ecosystem gaps to the Grant for the Web advisory committee to inform future grant making.

Together, we completed the following activities in support of our objectives:

Design a workflow that builds on the Web Monetization protocol

We completed the first iteration of this work during the research phase of the funded period. We made an early discovery that Coil was the only viable web monetization payer we could leverage for this work. We honestly could have spent months digging deeper into web monetization, but we knew that building the Etherpad integration would teach us more than static research, and we very much wanted to deliver a usable proof-of-concept.

Upgrade the Permanent API to support that workflow

We made critical upgrades to the login functionality of the Permanent API to achieve the goals of this grant. We added endpoints to our API that allows a 3rd party app to retrieve information about a user’s login status and their currently active archive as well as a mechanism to endow space to a user’s account equivalent to the Etherpad file size. The core mechanic here operates under the assumption that Etherpad files are very small and the Coil payment received from a user’s activity more than covers the cost of storage.

Build a simple demonstration application that uses the protocol to store content

The main tangible output of our work is a repository containing an Etherpad plugin that implements web-monetized (Coil-based) micropayments to authorize permanent storage of pad content. That repository is available on GitHub, and we published its contents under a GPLv3 license.

The plugin we built demonstrates the functionality we originally envisioned to connect a generalized web-monetized application for content creation (or curation) to a storage layer capable of accepting web monetized payments. The plugin contains generalized functionality that could be plugged into any Etherpad instance.

The initial prototype installation of this system that we demonstrated at the Grant for the Web community call in June was hosted at https://pad.staging.permanent.org. However, we chose to host the final version of the system at https://pad.opentechstrategies.com in order to demonstrate a separation between pad and storage provider.

While Etherpad does not require a user to create an account with the host, users will be required to create accounts with Coil and the storage provider in order to save out a pad.

Document everything

Full documentation for installing this plugin is available in the project Readme visible in our GitHub repository. We documented our work in a Permanent blog post, multiple web monetization posts (introduction, interim report, etherpad justification, ecosystem analysis, etc.), and in a slide presentation of ecosystem gaps to the Grant for the Web advisory committee.

Ecosystem Gaps

In the course of adopting Web Monetization, we began with an evaluation of Web Monetization. We came to respect its privacy characteristics, even as we considered how to give users some insight into the trustability of the system. We considered a number of payment structures one might build on top of Web Monetization and the difficulty in creating new models using the current infrastructure. As we studied and implemented, we came to understand that the Web Monetization ecosystem was missing some elements we wished existed.

Coil is well-suited to replace advertising revenue models. It produces advertising-like revenue, which has some characteristics that make it difficult to replace fee-for-service revenue models. In particular, it is difficult to know how much a user has paid. Permanent's pay-once model provides space to users in proportion to fees paid. Those fees are low relative to the long timeframe of service offered, which makes them well suited to micropayments.

Coil, by contrast, provides payment in proportion to service usage, and that usage is proportional to the time spent active on a page. Sites do not get instant feedback on which user paid which amounts, which is what they would need to enact fee-for-service models. A Coil-enabled website monetizes content and can enable premium features, but it has little control over how much it charges and has little ability to scale premium service to fees. That limits Permanent's ability to increase storage quotas reliably.

As a site owner choosing to include Coil as a payment option for our services, we were hamstrung by a lack of granular transaction data. Without the ability to know immediately who paid what amounts for what services, there are strict limits to the types of things we can do. Coil's approach to privacy and security simply does not allow it. This is a point in favor of that approach to privacy. And yet, it made Coil a difficult fit for our current business model.

That realization left Permanent with several options.

• We could abandon Coil and build our own web monetization payments infrastructure. That promised to be an insurmountable effort, much larger than the current project (but one we hope to see happen eventually).
• We could write our own browser extension and try to leak information to our server. This also would be a large amount of effort, and the goal of this project was never to make a less privacy-protecting Coil.
• We could look to a micropayments system that is not built on web monetization. This would take us beyond the scope of the project, but because our goals were research-related, we did examine other solutions. There are a number of them. None are mature, complete solutions that would enable us to serve users effectively and immediately.
• We could adjust how Permanent allocates storage for users in the face of inexact payment streams, essentially fitting our business model to the available infrastructure.

After examining each of these options, we chose to adjust Permanent's backend to work with users, even when we cannot tie payments directly to a specific user. We did the math on payments versus the amount of storage required to save Etherpads, and decided we would come out ahead. We plan to test usage over time and verify our numbers which were necessarily based on some assumptions.

This scheme works for small files like Etherpad pads but it breaks down as the file size increases.

Ultimately, we adjusted our business model to suit the existing Coil infrastructure. That's not ideal, as we ended up with worse product-market fit than our usual offerings. An ecosystem limited to Coil's offering of time-based payments decoupled from specific users is missing some necessary pieces. We wanted to see support for one-time-payments, streaming, and various forms of tipping. More broadly, we wanted to see bundled subscriptions that allow several services to make joint offerings.

In addition to those big-picture difficulties, we observed some minor missing elements of the ecosystem:

• Information for users. As currently constructed, a user who installs Coil has very little ability to see how it changes their online experience. Where did my $5 go? What sites did I support? What benefits did I receive? All of that is opaque, which might make Coil a difficult sell as a value proposition.
• We struggled during development from a lack of test infrastructure. We wanted mock wallets and accounts so we could demo all this end-to-end, dig into the details, and understand what was happening. Especially in a system where developers cannot see much information about transactions, we wanted a test environment with more inspection capabilities.
• The need to install a browser extension continues to be a barrier. Although many users have installed ad-blocking extensions, the vast majority of browser users have not. Ad-block penetration at around a quarter of all users has been relatively steady for several years. And once we step beyond ad-blocking, the numbers only go down. Coil has momentum and investment that could overcome reluctance to install extensions. The greatest chance for non-Coil web monetization solutions to reach wide adoption might be as an additional feature of Coil, not as a separate extension.

Authorization Upgrade

In order to make the etherpad plugin specifically and 3rd party web monetized applications generally compatible with the Permanent platform when hosted independently from the Permanent.org tech stack, we needed to upgrade our authentication framework.

Using the time granted by our no cost extension, we undertook a research inquiry to determine the best implementation option for managing authorization. We consulted with experts in the auth community, including D. Keith Casey, Jr. Based on our research we determined the best solution to be a 3rd party authentication provider and after evaluating Auth0 as an option, we selected FusionAuth as our vendor.

We published the work we did to migrate our authentication data to both Auth0 and FusionAuth in our GitHub repository.

• https://github.com/PermanentOrg/migrate-permanent-auth0
• https://github.com/PermanentOrg/migrate-permanent-fusionauth

Communications and marketing

Our communications and marketing efforts were focused on inviting others to help us build this solution together. We distributed our messages via the web monetization community forem, community calls, and advisory board meetings, as well as the Permanent.org member testing group, blog, knowledge base, social media and email newsletter. We also did all of this work in the open in our GitHub repo and documented our results there.

Through several posts on the Web Monetization Community forem, we shared project updates, insights into our work, discoveries about the web monetization ecosystem, and invitations to evaluate our prototype (see our project page). We joined a Grant for the Web community call on June 24, 2021 and provided our etherpad instance for note taking as a way to highlight and demonstrate the results of our work. It was an honor to have our prototype pad instance highlighted in this way.

Additionally, our project was selected to be featured on the Coil developer site and we participated in the Coil Web Monetization Workshop on July 28, 2021.

During the course of settling on Etherpad as our demo app, we also reached out to the folks building the amazing docs.plus tool! Our conversation with Edward Saperia was particularly inspiring and expanded our understanding of what was possible to build on top of Etherpad! We consulted with experts in the auth community, including D. Keith Casey, Jr.

Our project began and ended with user research. Before any design began, we conducted informal interviews with non tech-adjacent individuals in our community to understand how they would expect this kind of functionality to work for them. Once we had a prototype solution in place, we began to user test with our member community.

We communicated with our members about the etherpad plugin work using Permanent resources. We published this blog post and shared it [via social media]. We made a call for our members to provide feedback on our prototype in our monthly newsletter. And we created a dedicated knowledge base article to help folks understand how the prototype works. Now that we have a final product live on our production server, we are ready to take another shot at communication and marketing. We hope to add this as a prominent feature on our marketing site home page.

What’s next?

Our work on this grant has concluded, but we're not done with Etherpad, the plugin, or web monetization. We want to explore using our plugin with Docs.plus, another GFTW grantee that is already using Coil. One question we have about this path is how Coil works when a user is consuming services from multiple providers.

In addition, there are opportunities to help users better manage their archives of etherpad documents on Permanent.org. If we imagine adding permanence to etherpads across the internet, that's a lot of documents. At that scale, users might need tools that support sophisticated curation of etherpad collections.

We aimed this work at simple, MVP functionality. There are features we would like to add that allow portability of documents between pads, storage to multiple locations, and improved access controls.

More broadly, we could expand this approach to many different types of content, not just etherpad documents. In fact we have already started to explore other compelling use cases thanks to the progress made possible by this grant.

Our most compelling partnership in development today is with the Hyperaud.io team who are currently working on their own Grant for the Web funded project to integrate web monetization into their video transcription application. In this scenario, Permanent.org serves as both a storage layer as well as a streaming platform.

We’re also excited about work we’ve done with Theirstory.io as well as conversations we’ve started with Ponga.com and Webrecorder.org. Each of these applications demonstrates a unique use-case for a web-monetized content to storage workflow.

This work has pushed us toward considering how web monetization fits into Permanent’s overall vision for products and services.

What community support would benefit your project?

There are three key requests for our Web Monetization Community to help support this work:

1. Host it: If you host an Etherpad instance or are interested in hosting one, we would love to have our plugin installed on your instance.
2. Use it: keep your meeting notes in Etherpad and use our instance when you do, then save your pads out to Permanent.org to preserve them. If you are an application developer, we’d love to explore opportunities to integrate with our API.
3. Improve it: our Etherpad plugin is an open source project and we’re eager for members of this community to help us improve it.

Please feel free to reach out with any comments or questions you might have about how to participate.

Additional comments

We would like to thank the Grant for the Web team for creating this truly valuable opportunity to expand the capabilities of Permanent.org. We are so grateful to have been granted the time and resources to work on a unique project. We hope this work expands the capabilities of web-based applications to build a decentralized content distribution and storage ecosystem.