The Interledger Community 🌱: Flaki

Waasabi — Final Grant Report

Flaki — Mon, 09 Aug 2021 07:41:24 +0000

Project Update

It’s been a year since Waasabi's first prototype was conceived, so amidst the ongoing planning of our second large conference and onboarding of new early-adopters to Waasabi Beta it's time we discussed what's already done and what we sets ours sights on for the the upcoming updates.

Progress on objectives

It was November when last year the first version of Waasabi powered rustfest.global. Like many other events, we too were forced to migrate into cyberspace, and Waasabi was our way response to seeing the lack of creativity and missing tooling in the online event space. Since then we have been moving our learnings and code from that experimental version into an open and flexible framework under the Waasabi Framework's umbrella.

In the past months multiple events (Rust Berlin meetup, Rust Hungary meetup), even our own development stream the Waasabi Live stream has used Waasabi as we worked to constantly evolve and expand its feature set through this feedback.

The work is still far from done, though Grant for the Web has enabled much experimentation and laid the foundations upon which we may continue expanding the initial scope as we dive deeper into creating the tool that will enable the engaging interactive experiences we have envisioned.

All Waasabi components are open source, released under the Apache-2.0 license on Bay Area Tech Club's GitHub.

Key activities

Waasabi set out to

"solve the hard problems of event streaming"

and through this, to enable event organizers build custom experiences, quickly. The toolkit's underlying concept is to create a coupling of efficient live streaming and a variety of realtime integrations (such as chat, captions, interactive experiences and more). Waasabi is the "hub" that aggregates data from all these systems and provides a unified API, as well as real-time event notifications (e.g. of incoming messages or when a new live stream is started).

Instead of a single, long-running live video stream — which is how most online events today are usually structured — Waasabi encourages a single stream per program and will take care of starting & switching between streams automatically. It may even multiplex multiple simultaneous live streams, for example to allow showing the live stream and a live sketchnoter, side-by-side.

The conference live stream shown on the left with a live view at the sketchnoter’s work on the right at RustFest Global 2020

Waasabi Core Beta

Waasabi's Core module extends the open source Strapi content management system with its own preset template. Waasabi's template is used by the installer (see below) when creating a new Waasabi instance from scratch.

The Strapi administration interface of the instance can be used to manage Waasabi's events and broadcasts. Strapi's API can be used from any existing website to integrate streaming functionality. The Waasabi Installer also provides a "Live Event starter kit" that can be used for the live experience and further customized if needed (more on this later).

The current Beta focuses on public events, while built-in registration, ticketing and user management has also been previewed at RustFest Global 2020, and is coming to the suite in the next version.

Web Monetization

Waasabi aims to be a collection of tools

"…for [event] organizers who want to experiment with new event formats and business models…"

It intends to encourage & help organizers fully embrace that online events come with a completely different set of limitations, when compared to in-person events, but also have their unique advantages. For example, people might be less focused and less present during day-long digital events: in-person events bring with them a very different headspace, that cannot be replicated from home. On the other hand, given enough flexibility and organizer support, features like Waasabi's instant replays, captioning and other affordances can make online events orders of magnitude more accessible, with a much larger, potentially global reach.

Web Monetization is a great example of “things that would not make sense in an in-person context, but can work really well for online events”

Web Monetization as a creator incentive

During RustFest we wanted to use Web Monetization for more than just a revenue source for the event: we wanted it to serve as an incentive for the true producers of the conference content, the speakers and artists.

The way this is currently implemented in Waasabi, is that every session can be assigned a Payment Pointer and Waasabi takes care of switching to the correct payment pointer during the session’s runtime. While this works, it’s a crude solution that raised multiple other issues:

What happens if a session has more than one speaker/collaborator?
Having all speakers and artists register & share their Web Monetization-enabled wallets was cumbersome and many missed the opportunity due to this friction;
Making sure the attendees had Web Monetization-enabled clients (subscription with the required extensions installed) resulted in even more friction.

With increased awareness and adoption some of the above issues could be alleviated, but we are also looking into Rafiki, and exploring hosting our own Interledger service within Waasabi, which could provide a lot more flexibility in these micropayment integrations and open up more creative ways to utilize Web Monetization in Waasabi with greatly reduced friction.

Web Monetization for access control

Initially we had planned to enable using Web Monetization as an access control mechanism: gating some of the content, or enhancements such as higher resolution streaming behind streaming payments. Watching the evolution of online community conferences, it quickly became clear that:

in order to have the best possible, global reach and make community events accessible to all a free event or free tier is needed;
in fact, many seem to came to expect online events to be free, with costs covered mainly by corporate sponsorships;
but one can still capture funds via “ticket” sales by providing extra features, especially when employees can forward these costs to their employers.

Due to these, Web Monetization access control to these additional features or the stream itself became a lower priority and this first Beta focuses on publicly available events.

We are looking at potentially implementing video access control and monetization via Peertube, a new video backend we integrate with. Peertube is receiving first-class Web Monetization support via another GftW grantee’s project and making this available within Waasabi is something we are looking into.

Configuring Waasabi instances

Waasabi’s goal is to enable everyone create their own custom digital event experience. This means we need find ways to allow people create highly customized instances relatively easily.

Our initial approach focuses on the Waasabi Installer, a cross-platform command line tool focusing on the more technologically adept users of Waasabi. This tool allows configuring all of Waasabi’s features and generating a new instance, that can be ran locally for testing or deployed directly on a server.

Currently direct server/cloud deploys are not implemented yet, but the configuration files generated by this tool makes it rather trivial to run Waasabi on any cloud or webserver that is based on Debian, or one that at least can run Ubuntu containers or virtual machines.

Waasabi and the Installer runs well in WSL, so Windows users can also configure and test Waasabi instances before deployment.

Customizable event page starter

As noted before, Waasabi’s core provides an API to embed the live streaming functionality directly into one’s existing website. We plan to provide more “batteries-included” tools for this in the future (such as iframe-based integrations, and plugins for popular website tools, such as Wordpress), but currently this needs to be done manually.

To make Waasabi usable even without the need for lengthy manual integration, we provide an out-of-box experience with the Waasabi Live Page. This interface can be configured under a subdomain of the existing website and provides complete functionality of Waasabi: streaming, Web Monetization, chat integrations and more. Some in-development upcoming features such as registered user management are coming soon as well.

The Live Page can be configured from the Installer, and its appearance can be further customized via “brand packages” that may override any of the content, CSS, images and other assets, and may even implement additional functionality via client-side JavaScript.

Chat support

One of Waasabi’s fundamental goals was stated to be:

"[allow organizers to] integrate their live events into their existing community infrastructure"

Waasabi explicitly opts for not reinventing a wheel, and does not provide a built-in chat within the live stream page itself, but instead chooses to integrate with existing chat providers. Many communities who do live streaming already have an existing community chat which they often end up wanting to integrate with their live streaming platform of choice anyway, so Waasabi circumvents this by going the other way right off the bat.

Multiple integrations are planned for Waasabi, but the currently implemented one in the beta builds on the robust and fully open source Matrix network. Currently this means messages sent on the linked chat rooms are relayed to the live stream.

Once the work-in-progress account management pieces land users will be able to join private channels from the Waasabi interface. This feature was initially used at RustFest Global, and allowed attendees to be added to specific conference channels automatically or on request. Ensuring proper access control of the event chat is really important (especially for publicly accessible live events), and allows hosts to perform better moderation, enforcement of the event’s code of conduct and keeping the conference chat safe for all participants.

As noted, we are still looking into integrating with more chat services, especially looking at Discord and Slack to allow event hosts to bring their existing community into their events, as well as use their events (meetups, streams, etc.) as a funnel to bring more community members onto their community chat of choice.

Ownership & control

Waasabi tries to offer as much flexibility on the axes of content ownership and full control as possible. Administrators of the instances can configure Waasabi with the Mux.com streaming backend provider for a reliable conventional streaming service, or choose Peertube and many other planned self-hosted video backends that fit their requirements. The framework allows for making these decisions in accordance with one’s requirements, expectations, scaling needs and other individual needs.

Realtime captions

We are also experimenting with tools for automated captioning, improved transcripts and similar accessibility improvements. You may learn more about our experimental work as part of this grant to bring machine learning and speech recognition to Waasabi without third-party dependencies by reading our last detailed report!

Complete control over content

Waasabi exposes the low-level building blocks of live streaming. This has already made functionality like instant replays and web monetization possible, and we are constantly experimenting how could we use these possibilities in novel, unconventional ways.

Waasabi is also embracing various peer-to-peer technologies in a form of another grant. The Peertube video backend is one of the first early results of this effort. We strongly believe peer-to-peer technologies allow for better positioning these self-owned and individually controlled streaming hubs, further improving discovery, robustness and reach of the instances, allowing Waasabi to provide a viable alternative to large corporate silos.

Extensibility

In the name of unparalleled flexibility we set out to encourage a Waasabi ecosystem of

“[…] a growing library of open source plugins and integrations
contributed back by the community building on Waasabi”

There is no point of pushing for plugin development before the stabilization and proper documentation of Waasabi’s API, but once these pieces fall into place we are looking forward to people building new integrations and contributing back to the commons. Until then, of course, Waasabi instances can be customized in the ways mentioned further above, and of course bugfixes and feature contributions to Waasabi’s open source codebase are appreciated and encouraged as well.

Communications and marketing

Waasabi’s website has recently went live and is continuously being expanded with documentation, guides and updates. We are also waiting for Waasabi’s first branding to be delivered to dress the project into a brand new exciting and playful gown.

Waasabi’s development live stream is currently offline as it is being upgraded to the latest Waasabi version, and shall be returning with development updates in the coming weeks.

We are constantly looking for early adopters interested in experimenting with this early release of the Waasabi suite (and are in contact with few already), and we plan to bring back the RustFest conference this fall, this time around using the open source Waasabi suite instead of the experimental version from 2020.

We are ready to answer any questions and discuss Waasabi’s future in our dedicated Matrix channel at #waasabi:baytech.community.

What’s next?

The development of Waasabi continues, there are many things to look forward to!

Accounts and registrations (e.g. for ticketing)
An advanced event administration interface, provided by the “event-manager” Strapi plugin of the Waasabi template
Subscribe to events on the live page (email notifications, add-to-calendar, RSS and more)
P2P enhancements, distributed streaming and connecting to decentralized networks
Independent automatic captioning and transcript-augmentation via customizable machine learning detection
Further Web Monetization and Interledger-experiments using Rafiki

Some of these features above we plan to experiment with and test on our upcoming RustFest conference this fall, and other early-adopters.

What community support would benefit your project?

Interested or have questions? Join our Matrix channel and talk to us!
Want to run your own event? Please reach out, we’d love to help!
Like what we are doing? Consider supporting the project on Open Collective.
Find links to all of these channels below.

Relevant links/resources

Automated captions for everyone with Waasabi

Flaki — Fri, 23 Jul 2021 09:03:28 +0000

This is an update on the automatic captioning proof-of-concept that we have been working on for the past couple months, as part of our grant for the Waasabi live event framework.

Below you can read about our motivations, the progress we have made, sprinkled with some early demos and examples, as well as some of the future plans. This is going to be a long post, feel free to jump around to the parts you care about.

On Transcripts

The earliest motivation of this work was born out of our dedication to inclusive and accessible events at RustFest, and had nothing to do with automated captions. All our events in the past years - online, or in-person - have been live closed-captioned by a professional stenographer. These live captions have made our content accessible for disabled people and also attendees who were not native English-speakers (which, as you'd imagine for Europe's largest Rust conference meant most participants).

Our live captioners also produced a transcript of every talk, that we were hoping to put to good use eventually — but in reality for the longest part were just collecting dust tucked away in a GitHub repository. RustFest Global now publishes all talk transcripts on the individual session pages, which was an improvement in some ways (searchability, SEO), but at the end of the day this was still very far from the ideal user experience. Throughout the years we slowly recognized what we truly needed was a more direct connection between the audio, video and text.

The thing our transcripts were missing dearly is the timings. The difference between transcripts and captions is what's called "timecoding" in industry terms: annotating the transcript with time codes that correspond to precise timecodes of the original recording. We always made sure to allocate funds for captioning from the conference budget, but being a small community event it was never easy to justify the extra costs of hiring a professional timecoding service after the event already ended.

YouTube getting better at captioning has alleviated the pressing need for getting our transcripts in order (more on this later), but we never truly given up on making this happen without the big G watching over our shoulders.

Closed captions for conferences, today

A small intermission before we move on. There's no question whether paying (often thousands of dollars) for a professional live captioner's time is worth it, when it comes to the accessibility of a paid event — that is what conference budgets are for. That said (and this was true even before the Coronavirus-pandemic) flying a captioner and their equipment to the conference is not a cost that is worth it — for the organizers, and the captioner alike. Why yank someone out of their cozy home or office, away from their family, hurl them halfway across the planet amidst a climate crisis just to... spend a day listening to someone talking into a microphone and transcribing what they said all the same? Right?!

Well, of course it's not this easy, as latency becomes an issue. Live captions will have some inherent latency — even the fastest captioner needs to listen to, understand, and type up what is being said — but when you add VOIP, internet roundtrip latencies and other factors, on occasion, captions would sometimes become significantly delayed (in the tens-of-seconds range) which may very well become confusing, especially for abled people who can hear and understand the audio of the presentation.

As a weird twist, online conferences often exhibit the polar opposite of this issue! Due to inherent properties of the widely deployed HLS and similar streaming technologies, the so called "glass-to-glass" latency of live streams is frequently 15-30s or even higher. This forces event organizers to choose between two bad choices: stick to the traditional method of putting captions onto the streamed out video (and accepting the significant latency inherent to the work of a remote captioner), or push the captions directly into the clients on a separate channel, which risks desynchronization and potentially even captions arriving before the live stream catches up!

At RustFest we are not really fans of "the lesser of two evils"-sort of ultimatums so we had to come up with a plan. And so we did.

The Waasabi Captions PoC

Disclaimer: This is a proof-of-concept. This is not ready-to-use software, but a couple of focused experiments to establish the feasibility of this endeavor. The experiments showed quite promising results, and we intend to publish the current work as well as continue to grow this project into software that could be used by anyone (with or without Waasabi itself). If you like the direction please consider supporting the project or even contributing to the work (find the relevant links at the end of the article).

In keeping with the spirit of Waasabi, the goal for this captioning component was to provide a live captioning solution for event organizers that is completely independent of external services and API providers. Our focus is on small, largely free community events, organized primarily by enthusiasts, and not to cater for enterprise events and large, multiple-thousands-strong commercial conferences.

It was clear that we should be looking for a solution in the booming machine learning space, and importantly, one that used free & open source software and royalty-free training data, available to anyone. This proof-of-concept integrates Mozilla's DeepSpeech project for machine-learning-assisted speech-to-text processing, which itself is built on models generated from open training data using the industry-standard open source TensorFlow framework.

Testdriving the default model

The experiments below are using Sean Chen's talk on Rust Error Messages from last year's online RustFest Global conference as an input. Sean is a fairly eloquent native English speaker so we are playing easymode here, but for a proof of concept this serves as a sufficient foundation.

First we fed the unmodified talk audio into DeepSpeech's pre-trained model — the result of this was already really interesting!

As part of this exploratory research we have created several, relatively simple JavaScript tools to aid us in the evaluation. One of these tools takes DeepSpeech's output and tries to cross-reference it with our existing transcript for the same audio. Another tool is a visualizer, that takes the output of this "stitcher" and shows it visually in a HTML form, this is what is linked below:

Comparing the DeepSpeech output to the existing transcript

On one hand, the ~60% match rate is nothing to call home about. Trying to build a fully automated captioning system without further finetuning would yield rather unconvincing results.

On the other hand though, when we take a closer look, we realize that the accuracy of "regular" talk transcription is actually reasonably good!

Now, it is nigh impossible to compete with the tens of millions of people using Alexa or Google Home every day, or the multiple years worth of content uploaded to YouTube every hour — the voice data and resources that are at the disposal of Amazon, Google and the likes are astounding. So the fact that we could achieve decent transcription without mountains of data and datacenter-sized compute resources is already pretty amazing — but we also have at least one advantage over these megacorps... but well, let's not get ahead of ourselves.

Remember, this is the pre-trained generic out-of-the-box model that ships with DeepSpeech releases, let's see what possibilities can some customization unlock for us...

Further fine-tuning

Without wading too deep into how DeepSeech works, there are two main components: the main beam search decoder, a neural network that transcribes sounds into characters of an alphabet and constructs the output words alongside an (optional) external scorer that helps to augment the results. The scorer contains a language model (generated from a large collection of text from the target language) and helps "bias" towards word compositions that are more probable. Here's a simplified example of the concept:

NEW followed by YOLK is not really frequently used in the text corpus our scorer was built on. But NEW + YORK is, and especially when followed by CITY. Our scorer will use this knowledge to influence the final result of the decoding accordingly.

There are, thus, two ways to improve transcription accuracy:

improve the acoustic model — to transcribe the spoken audio more accurately, and
improve the scorer — build a better language model to better bias towards more relevant word combinations

The first can be accomplished via fine-tuning, an exercise for after the proof-of-concept, but below we will be talking briefly about early results with the second approach.

We can generate a new language model for DeepSpeech using any text corpus — any sufficiently large body of target-language text will do. In a generic scenario, we would need to be mindful of overfitting, creating a model that is only any good predicting the data it has been trained on, but... well for timecoding, that is precisely what we need! We can generate a custom language model from the collected RustFest talk transcripts and see how it changes the equation:

Timecoding the transcript with the overfitted custom language model

Whoah, look at all that beautiful bright green! Of course the transcription is still not perfect, there are two main sources of error that remain still:

Some of these are "mishearings" of the acoustic model, that cannot be corrected even with such an overfitted language model, but
Some of these stem from the fact that human captioners take certain liberties when they transcribe a speaker: some of these are genuine and improve the brevity and/or clarity of the transcript, and occasionally these come from omissions (e.g. when a transcriber falls behind).

We will share some thoughts about how we plan to account for these issues going forward, but first, let's see what we can do with this data.

Using the DeepSpeech results

Okay, so DeepSpeech spit out some text & timings and we correlated this to the transcripts, what now? Taking over the world, of course!

For example, we can easily turn a time-coded transcript into subtitles!

We are currently in the process of publishing all previous RustFest recordings on our website and this allows us to provide subtitles for all talks, as well as a transcript. In fact, we can even allow the user to jump to parts of the video by clicking on the transcript — similarly how the visualizer allows for jumping to parts of the talk when clicking the. However all this is just the tip of the iceberg.

So far we have only used DeepSpeech on existing recordings, but DS is a streaming speech-to-text engine, it supports real-time inference! This means it's possible to do speech to text on the Waasabi backend and pipe the captions out in real time for any incoming video stream:

Streaming the live captions into a Waasabi event

The above would allow low-budget or no-budget events (such as community meetups or similar) to provide completely automatic "best-effort" captions to any content using fine-tuned, custom models that fit their purpose, but this could even help events with human captioners improve their user experience, too! How?

We could turn the livestream's latency to our advantage! By having the human captioner in the "studio" (near-realtime access to conference feed) they would be "ahead" of the stream as they were transcribing it. Meanwhile, we would generate an automatic transcript as well, timecoded to the currently active live stream. By combining these two in the same way we have demonstrated above, and sending down the generated captions on a direct realtime channel to the clients (e.g. WebSockets), we could ensure the captions appear exactly when they were supposed to during the client's playback! No more spoilers!

Artisanal, small-batch, hand crafted machine learning

As we said before, it's impossible to compete with data, resources and sheer scale of Google and the likes as an idividual or small community. We ourselves have resorted to be using YouTube's captioning and transcript-auto-syncing (around which seemingly an entire cottage-industry of "how to download your YouTube subtitles" content producers has sprung up). We do have one advantage though: in the sea of all this content, YouTube remains generic, while we can differentiate and bias towards our own content.

As showcased before, a lot of the words the pre-trained (generic!) model missed — Rust, errors, compilation — were the topic, or even industry-specific ones. We can afford to bias our models towards these without having to worry about other users, other consumers. What's more, our own models, trained on historical Rust content may be used by other events: conferences, meetups, live streamers! Then, these content producers would reap the gains of this focus for their own content and audience, after which they, too, could contribute their own recordings, transcripts to be used to further fine-tune the model and contribute back to the "commons"!

So what's next?

We are coming away with really promising results from our research on this proof-of-concept and are eager to start the 'productization' process, to be able to use these techniques and technologies on our own upcoming events — and can't wait to see others test-drive and adopt them.

One of the first things on our list of "next steps" is creating a self-contained RTMP stream proxy, that event hosts could pipe their live studio directly into for real time deepspeech transcription. The tool would take care of pushing the transcribed speech content into Waasabi, as well as forwarding the studio content to its intended streaming destination (Mux, Peertube, or even any other destination outside of Waasabi).

Another very exciting feature we are looking into is in the front-end side: implementing crowd-sourced suggestions into the system. This would allow people watching the event live stream, replays or published recordings to suggest changes to the transcript in real time which could greatly improve the precision of transcripts. Since Waasabi provides instant replays for event attendees, making it possible to correct transcripts on the fly would provide a huge boost to the user experience of viewers joining in later on.

We are also currently experimenting with employing the precisely time-coded transcripts to automatically fine-tune the acoustic model. In theory, we could have DeepSpeech and some clever scripts identify parts of the document that the current model misses use the audio extracted for these sections and the transcript to propose targeted fine-tuning of the acoustic model. This would especially become interesting for speakers of various English accents to make the model more capable in detecting and transcribing non-vanilla English speakers.

Finally, we are also looking to working with other communities and content producers as early-adopters of this technology. We are also looking at adjacent content-producers, such as podcasts with a large archive to see if they can make use of this technique to augment their podcast archives and provide live transcription.

Let's stay in touch!

Thank you for your interest in this long report, and we'd like to thank Grant for the Web for funding this effort!

Follow our updates on Open Collective and Twitter. You will find all the source code on GitHub. You can ask questions and chat with us on Matrix.

Waasabi development update: Chat Integrations

Flaki — Sat, 27 Mar 2021 13:59:47 +0000

Quick update after field-testing one of Waasabi's upcoming chat integration feature on Monday's Berline.rs Meetup!

The Waasabi framework intentionally leaves the well known "stream chat" functionality out of the package. The reason for this is not just the sheer complexity of implementing an accessible and abuse-resistant chat client, but is also based on the observation that most communities in Waasabi's target group will already have a chat system they were be using for the community and integrating these would be preferable than a wholly new, separate system just for the chat accompanying the stream.

At the very first RustFest Global pre-event: @dns2utf8 introduces us at Rust Zürichsee to playing self-isolating snake games—together! (also test-driving our new conference experience platform, stay tuned for more!)

Join the fun via Matrix: #rust-zuerisee:https://t.co/BelQToQ9VW pic.twitter.com/LAniZ9GLeA
— RustFest (@RustFest) November 2, 2020

An early prototype of Waasabi's chat integration from RustFest Global 2020

Access control and moderation of the stream chat is also an ever-present issue: to prevent abuse, some kind of access control for the chat is desirable for most communities. With existing event platforms, we have often found support for this lacking and reek with privacy compromises. That said, there is clear vlaue in providing frictionless access to the discussion even to those who may not want to join the chat channels for the broadcast: we accomplish this by displaying the ongoing discussion in a read-only fashion on the stream. As it is often with these things, this ends up being a balancing act and matter of personal preference, and so we are still experimenting with the best approach to exposing this feature in the user interface.

Chat messages are relayed in real time and display on the Waasabi page

For every community out there, there is a plethora of community platforms to choose from. Waasabi's chat support thus has been designed with a high degree of modularity and freedom in mind: by exposing a generalized Chat API that accepts new chat messages over a regular HTTP endpoint we can support various services through chat integration plugins. These plugins are usually external services that act as bridges between the network in question and Waasabi's backend, which takes care of forwarding incoming messages to clients following the active broadcast.

Waasabi's first such bridge integrates with the Matrix network. The plugin is written in the Rust programming language, and relies heavily on the Ruma framework for communicating with the Matrix network. Developed by Jan-Erik Rediger to support the RustFest Global event in 2020, the Matrix integration is currently undergoing work to bring its full feature set, such as private invites and handling of multiple rooms to Waasabi for the upcoming first public release.

Other integrations are also planned, Discord and Slack are two networks we are actively looking into bridging early on, but thanks to the open nature of Waasabi's API, it should be relatively easy for adopters of the framework to bridge any other network to their instance, even if Waasabi may not have out-of-the-box support for them.

Waasabi live event framework — Grant Report #1

Flaki — Thu, 11 Mar 2021 14:51:00 +0000

Waasabi's idea first formed in March 2020. As an avid conference-goer and organizer I, too, like many others, was disillusioned. Most events that have "moved online" felt like giant square expo-center-pegs, stuffed into a round internet-hole, mere pale digital reflections of a bygone era of physical attendance.

Soon enough I realized much of this was not even for lack of trying: as we ourselves first embarked on the journey of bringing about a much remote, very online RustFest experience I had to face first-hand the frustrating lack of tools for creating unique, inclusive, engaging online event experiences.

The task at hand was clear: we had to build it.

After some experimentation by late spring an early prototype provided the outline to what later that fall was to become the live streaming experience of RustFest Global: one of the largest conferences for the Rust programming language, and one truly spanning the globe, with speakers and artists spread across multiple continents, languages and timezones.

The Waasabi prototype used to power the RustFest Global online conference stream experience

Project Update

Receiving a grant from Grant for the Web meant not only that we could experiment with new, unconventional funding and reward models built around Web Monetization, but has also given us a chance to continue the work well after the conference, turning the prototype powering RustFest Global's live experience into a tool that may be used by anyone and for a much wider range of use cases.

This transition is going well and we are gearing up to publishing the first edition of Waasabi, with continuous updates to follow afterwards.

Progress on objectives

In the past months tremendous progress has been made on executing on the project's initial goals. Below we will illustrate some of this progress in the context of the original objectives:

Waasabi set out to…

◦ “Solve the "hard problems" of event streaming and allow event organizers to build custom experiences around this core”

Almost any event that tries to create their own experience will eventually face implementing video broadcasts in one way or another. Waasabi, as you will notice, is wont to stray from the "tried and true", and video streaming is a good example of this.

Waasabi supports (encourages, even!) splitting up long events into short, per-session live streams. This small change in approach has surprisingly broad implications to the attendee experience! On the one hand, it allows for immediate publishing individual sessions for on-demand viewing (we simply call these "Replays") right after the live session has concluded — this makes events much more inclusive, especially on a global scale, as everyone can "adjust" the conference to their own schedules, instead of aligning with whatever the broadcast schedule mandates.

Ever wished for a conference where any & all sessions show up *instantly* in a Replay box after they aired, to be re(?)watched anytime, even if you just grabbed your ticket halfway through the event?

I did.

So I built it. 🙈

Probably my favorite @RustFest Global feature. 💖 https://t.co/X2sUhso2c4
— flaki (@slsoftworks) November 7, 2020

Another interesting implication of this change is that the breaks between sessions become a space we can fill with valuable, even interactive content. Whether it's generative art, interactive sponsor experiences or an embedded experience dedicated for connecting and mingling, the live stream gives way for other custom experiences, reclaiming its rightful prime real estate on the user's screen once a new live broadcast is due.

At the very first RustFest Global pre-event: @dns2utf8 introduces us at Rust Zürichsee to playing self-isolating snake games—together! (also test-driving our new conference experience platform, stay tuned for more!)

Join the fun via Matrix: #rust-zuerisee:https://t.co/BelQToQ9VW pic.twitter.com/LAniZ9GLeA
— RustFest (@RustFest) November 2, 2020

◦ “Provide flexible, extensible tools for organizers who want to experiment with alternative formats and business models”

Most commercial event platforms on the market today provide ready-made "just-add-water" experiences out of the box today. There is very little room for customization, integrations, or ways to engage with one's audience beyond what was pre-imagined by the platform's UX & design teams.

Another large chunk of events, who decided to simply opt for broadcasting their event on one of the large video streaming platforms are similarly limited: both YouTube and Twitch, two predominant providers in the space allow for very little access control over their broadcast, and do their utmost to keep revenue sources limited to and dependent on their own silos (ad revenue, subscriptions, etc.).

In either case, events have very little control over how their broadcast is presented, and who has access to it, thus it is pretty much impossible to freely experiment with a novel combination of micropayments, ticketing, sponsorships etc.

The RustFest Global live stream design subtly shifts in realtime to reveal one of the sponsors ahead of an upcoming talk

Retaining full control over the streaming experience also meant that we could offer interesting alternate ways to showcase our sponsors and allow them to engage in more meaningful ways with our audience — something that would be impossible in any of the conventional platforms. This meant that we have been able to provide value (and on this, we have received unequivocally positive feedback) to sponsors while keeping these appearances unobtrusive (e.g. instead of stuffing the live stream & recording full of distracting logos), which has been a key goal of our community-centered conference since its inception.

◦ “Enable increased reach through more sophisticated control over content distribution and revenue sources”

Having full control of every aspect of the video streaming experience allowed us to set our own rules, even provide free & pay-what-you-can style ticket options, while still retaining appealing incentives to those who would support the conference by buying tickets. That said, when it came to Web Monetization, we had a different issue in mind that we set out to solve.

Many events today support and incentivize content submissions by acknowledging the work put into creating a talk and the effort spent on educating the community. At RustFest we paid all our artists, given stipends and video streaming hardware reimbursements to our speakers in an effort to lower the barrier of entry for participation and sharing one's art and knowledge with the community.

Very happy to have had the opportunity to speak at @RustFest as a newbie speaker - this morning was a great experience :)
— Vi (@gl4cierBlue) November 7, 2020

We decided to use Web Monetization as yet another tool that would allow the community to give back to these artists and speakers, even long after the conference is over.

Every session at RustFest could provide its own ILP payment pointer (set by the session host or administrators) which would be active during the live broadcast of the session, as well as during any on-demand playback. This way, Web Monetization supported the creator directly, while RustFest merely provided the infrastructure and the "marketplace" to connect presenters with their audience.

Work on some of the initial objectives is still ongoing, see more information on these in the "What's next" section, as well as expect to see further updates on their progress in the coming weeks.

Key activities

Field-testing Waasabi's ideas in a realistic setting was one of our main tasks for 2020, and the RustFest Global conference provided fertile grounds for experimentation. After the conference ended, we took the distilled learnings and the prototype to create a more streamlined, more flexible tool that can be customized to fit one's own usecase.

RustFest Global

We believe RustFest Global was the first event that took advantage of Web Monetization in the online conference space. RustFest's approach was using Web Monetization as a secondary income stream for artists & presenters at the event, and on the long tail. With close to 1.000 signups, RustFest streamed over 4.000 hours of content & produced 15 hours of talks and art performances.

For most of our presenters, Web Monetization was completely new, and had a great reception, especially among our artists, with ~50% signing up and registering a wallet to monetize their content. Even after the conference, recordings published on Cinnamon Video, as well as those watched on the conference website would remain monetized towards the creator to provide a long tail revenue to the creator. To this date, our Cinnamon channel has seen over 600 views across all published talks and art performances.

In the post-conference survey the artists were also expressing their enthusiasm towards Web Monetization and were interested in using it in their future appearances and on their own websites, which further reinforced our suspicion that Web Monetization was especially well-suited to cater for content producers and performers in the virtual event space.

Continued adoption

Since RustFest we have spent the past months with streamlining Waasabi's offering and working slowly towards the public release & open sourcing of the first components under Bay Area Tech Club's umbrella. The various individual components of Waasabi will be released under an Apache 2.0 license, awaiting contributions on Bay Area Tech Club's GitHub page.

The components are written in a wide range of technologies — from HTML/CSS/JavaScript to Rust and a variety of frontend and backend frameworks. While for a company with a specialized team such a varied tech stack could very well be a nightmare-scenario to maintain, it is absolutely intentional here: we believe this allows contributors to pick & choose to work on parts of the Waasabi suite that they feel most comfortable in.

Rust Hungary's online community meetup hosted on Waasabi

While Waasabi will be sure to power any upcoming editions of RustFest, we have also started reaching out to the first early adopters to help us test-drive the suite's offering. We are working with these communities, meetups and conferences to ensure the out-of-the box offering supports a wide range of use cases, and focusing on improving the maintainability of instances. This means both streamlining the initial configuration, as well as making sure keeping them up-to-date is as painless as possible.

Work is well underway on waasabi-init, a command line tool dedicated to provisioning a brand new Waasabi configuration, either for local development, or one that is intended for a live installation. This tool will be responsible not only for applying customizations, but selecting, installing and keeping the various components of the Waasabi suite up to date. Much of our current efforts are channeled into improving the usability and lowering the barrier of entry of a Waasabi-based setup by abstracting away much of the underlying complexity.

Waasabi's configurator in action

If this blog post has piqued your interest in adopting Waasabi for your community, please reach out in the contacts at the end of the article, we would love to hear from you!

Communications and marketing

While most of the work so far has taken place behind the scenes, in anticipation of the upcoming first release we have started live streaming Waasabi development — on Waasabi itself! This may unarguably feel a bit meta, but is giving us an excellent opportunity to eat our own dogfood, as well as being a useful venue for raising awareness and gathering feedback. Currently the streams are a bit ad-hoc (follow the Waasabi Matrix channel for when to look), but we should settle in for a nice cadence once we have worked out some of the kinks.

Join the Waasabi Dev Streams at live.waasabi.org!

The Waasabi Dev Stream in action!

What’s next?

We haven't yet talked about external integrations and optional components ("plugins"), so let's have a look at what's coming to Waasabi in these areas:

◦ “Allow event organizers to tightly integrate their live events into their existing community infrastructure”

Just weeks ago the Matrix team debuted the official experience they envisioned for using the federated Matrix protocol in conjunction with online events.

Months ahead of this, in November last year we also used the Matrix network to power RustFest Global's chat experience through our own custom integration and our own RustCh.at Matrix server instance.

We are working on including a cleaned up & streamlined version of this integration in the components offered by Waasabi for those who would like to use this chat system with their custom streaming experience. We are also working on plugins for other popular chat platforms, to make it even easier to connect communities across Waasabi's interface simply by selecting and configuring the right integration, available out of the box.

◦ “A growing library of open source plugins and integrations contributed back by the community building on Waasabi”

While releasing & open sourcing the Waasabi codebase will need to take place before we can start talking about community-contributed extensions to the Waasabi software suite, we have already started laying the foundations to this through spending considerable effort on ensuring a low barrier of entry to trying, installing, using and contributing to Waasabi.

One of Waasabi's planned optional components was a "proof-of-concept" to a local transcription/captioning service, based in open data and open source software. We thought sharing the findings of this effort will deserve its own update, and thus keep an eye out for a coming update on this project in the following weeks. Check out a little teaser below of what we have in the works:

A comparison between a human-generated talk transcript and one generated by our automated local captioning proof-of-concept from RustFest Global

Future development

We are also thrilled to announce that thanks to the NLnet Foundation Waasabi has been recently awarded a grant from the NGI Zero initiative!

This project is still the planning/roadmapping phase but thanks to the new grant we will soon be able to start working on integrating P2P (peer-to-peer) video streaming backends as an alternative to Waasabi's current, commercial provider. We look forward to exploring technologies like HLS-over-IPFS and the recently announced PeerTube live streaming support.

What community support would benefit your project?

If the project's goals resonated with you, please consider supporting development through our Open Collective page.

If you would like to test-drive Waasabi for one of your own communities or events ahead of the public release, please let us know through any of the contacts listed below, we are always looking for feedback and early adopters who can help us improve Waasabi.

Relevant links/resources

Updated 2021.03.12: an earlier version of the post stated RustFest streamed 45.000 hours of content, this was incorrect, the correct number is around 4.400 hours, which is still half a year worth of RustFest!