The Interledger Community 🌱: Carolyn Malachi

Final Report

Carolyn Malachi — Wed, 01 Mar 2023 18:42:38 +0000

OVERVIEW

The Call & Response system (U.S. Patent Publication No. US-2022-0122570-A1) addresses two problems for individuals (referred to hereafter as “performers”) operating in the virtual performance space: payment friction and inaudible audiences. A matter made urgent during the height of the COVID-19 pandemic for music professionals who found themselves involuntarily out of normal work, also proved consequential for recording engineers, producers, songwriters, music instructors, and other professionals whose work shifted from in-person and relationship dependent to online and anonymously associated. Call & Response sought to and digital signal processing technologies to achieve its goal while preserving the privacy of audience members using only information from their live-virtual event feedback (comments, emoji, etc.) to guide what the performer or practitioner hears and how much digital currency that person or entity receives.

MAX / MSP patch converting incoming live Twitch audience responses into audible audience feedback

RESEARCH

When designing Call & Response, the question of whom the system needed to reach and if they would be likely to adopt the system once complete was raised. Women, young workers, and racial and ethnic minorities were among the major demographic groups experiencing the greatest increases in unemployment during the pandemic. Studies show that prior to the pandemic, women who worked in the music industry were already behind the economic curve before the pandemic with 90% working multiple jobs to earn a livable wage. Also, among the music industry’s studied population of producers and engineers, reportedly only 3% are women, a figure that is oft refuted by the population itself. Women working in the music industry appeared poised to benefit the most from the Call & Response system, especially given their daily interaction with technology. Neither of the available quantitative analyses included information about open payments or whether this demographic had used blockchain-based technologies to support themselves economically.

To acquire this information, we conducted qualitative research, traveling to Los Angeles, New York, and Nashville to interview some of the industry’s top producers and engineers. Our interviews centered around their pandemic-dictated changes to work and their comfort with new payments technologies. More than 75% of respondents reported hearing of such technologies but perceptions of illegality and complicated process dissuaded them. These interviews were critical to developing an understanding of how the technology should be presented to the target demographic. Interview subjects gave the sense that this technology is for other people, and not for them. How could this technology serve this group if they were not interested in it or were unable to identify with people they perceive to be the main users?

To hear excerpts of our qualitative research, click here.

Stated perceptions of users of this technology centered around ideas that current users adopt this technology to participate in illegal activity, or that the technology was too advanced, or that pain was an inevitability of growing to learn the technology, albeit full of promise. We understood that education, at the end of project development, would be a critical support to the marketing efforts being designed. We set aside 20% of our budget, $20,000, in a BlockFi interest bearing account, to prepare to support that effort.

DEVELOPMENT

Call & Response (patent pending number) combines two systems, both governed by an API call that extracts real-time data sets containing the verbal (text) responses from viewers of live streamed digital content. The incoming data sets trigger the playback of audience audio samples designed to model the surface properties and acoustic parameters of known performance environments, allowing the performer to hear once silenced virtual audiences as they would in their venue of choice. Simultaneously, the incoming information, according to the original Call & Response design, maps to certain parameters in a smart contract. The information, depending on the range in which it falls, triggers the release of predetermined amounts of a digital asset from one party (virtual event sponsor or promoter) to another (virtual performer / producer / instructor, etc.). The open payments layer of the system, the smart contract, would enable concert supporters to provide compensation in any currency while the performer would be paid out in the currency of their choice.

Strategy

Understanding the existing hesitancy among our focus population, we sought to leverage Twitch’s relationship with Coil. Doing so would allow us to model the web monetization feature of the Call & Response system using a well-known platform which would automatically, through its Coil integration, facilitate the delivery of digital assets (timely micro payments) to the virtual performer. We collected data (commentary, time of commentary, and username) from the live feed of the video. These bits of information were organized into a spreadsheet which fed the inputs of a sonification patch built in MAX / MSP. Inflowing data automated the playback of audience audio samples. Data thresholds determined which audio samples were played and made audible for the performer.

Expenses

With a process outlined, marketing professionals were hired for a five-month contract with the goal of developing a roll-out plan which included product testing among the designated target population. Based in Charlotte, NC, the firm is founded and operated by Black women. In addition to market research conducted to understand our target audience, we hired student interns to work on sound design for the project. Their work was supported by the purchase of computers, software, physical signal (outboard) processing gear, and other audio production equipment. This project also hired management staff and two developers. The first 80% of the grant payment covered the aforementioned expenses. With the expectation of moving into project testing at the end of 2022, the remaining 20% payment was held in a interest account with the goal of earning interest from the staking of USDC. Those funds remain in BlockFi’s custody through its ongoing Bankruptcy hearing.

CHALLENGES

The Fintech landscape evolved dramatically during the grant period. Changes to government regulation at the state level, custodian bankruptcy, and partner closing impacted our ability to execute the project as planned. On July 13, 2022, the Department of Labor in the State of Maryland updated its financial regulations governing money transmitters to include individuals or organizations facilitating virtual currency exchange and trading services. Call & Response is subject to this regulatory action, as the design is meant to do just that. The license application fee is nominal. Yet, the new law establishes that money transmitters, individuals and organizations engaged in the activity of currency exchange, including virtual currency exchanging and trading services, hold $150,000 of surety bonds. The state’s Fintech Innovation Contact exists to assist with navigating the compliance process, but the bond requirement exists nonetheless. The Interledger Foundation team, understanding of the complicated regulatory issue, suggested creating a model framework for the system, an entirely feasible strategy at the time. Also, an Interledger community member, Dr. Andrew Mangle at Bowie State University provided other suggestions for obtaining the necessary support to move forward with operation.

July 13, 2022: State of Maryland money transmitter regulation includes entities engaging in the exchange of virtual currencies

July 13, 2022: State of Maryland's Money Transmitter License application requires a $150,000 bond for each of the currencies supported by Call & Response.

Without the funding required by the State of Maryland to obtain a Money Transmitter license, we decided to leverage Coil’s products to demonstrate how incoming data from live virtual performance could initiate open payments for performers. We knew Coil as an established entity that could handle the transfer of digital assets while we developed the code to execute the sonification component. Testers of the Call & Response system would sign up for Coil accounts during the testing period to both experience the sonification and relatively real-time payments resulting from the flow of data from their live Twitch sessions. Be that as it may, Coil announced in January that it would sunset its products and development. It would no longer accept sign-ups from new users as of February 2, and it would discontinue its services altogether on March 15. This news derailed our testing plan.

Feb 2, 2023: Coil sunsets its products and prevented new users from signing up, thereby derailing our intended workaround for testing

Finally, with the first grant remittance mostly consumed by research and marketing campaign design, we felt it important to preserve for market testing the remaining grant funds which were paid after the first progress report. BlockFi had a strong reputation for safe storage as well as fair staking. We decided to hold and grow this $20k as USDC in a BlockFi interest account. Since then, then downfall of FTX and Alameda have been well documented, and BlockFi’s bankruptcy case thwarted by delays. Our budget for market testing remains tied up in the BlockFi interest account as the company has since November paused withdrawals.

November 11, 2022: BlockFi pauses user withdrawals as a result of the FTX / Alameda collapse. Education and testing budget still held there.

INSIGHTS

First, an opportunity for community education persists. Perceptions of illegality and technical complication create great apprehension toward open networks, digital assets, and other developments in financial technologies that expand access to financial resources beyond conventional financial services. We are developing a podcast that focuses on our target audience’s pandemic-era economic changes and their feelings toward fintech. It’s an opportunity to mitigate misinformation through community dialogue.

The current design of Call & Response necessitates public education not only on the subject of open payments and network interoperability, it also requires the user to understand visual programming languages and how to operate those platforms. The sonification component of Call & Response is conducted by a MAX/MSP patch. When opened, the patch automatically loads the spreadsheet to which live event data is stored, and that data triggers the playback of audience audio samples. However, programmed differently, with other audio samples, the output of the patch provides for the performer musical elements which could be incorporated in or released as new musical compositions. The performer then has an opportunity to, as they would with other work, generate sustaining income from royalties.

From one perspective, Call & Response in its current state successfully converts incoming data from live virtual events into audible audience responses which are pre-recorded audio samples. Data thresholds govern the timing of sample playback, and performer preferences dictate the acoustic properties applied to samples. From another perspective, the project also successfully provides as a tool for income generation the results of the sonification which an artist can place in the existing digital music market or the NFT market allowing for participation in open payments networks where the performer can set their own market rate and experience transparent payments, which differs from the traditional royalties system. Where Call & Response needs to make up ground is in its strategy for enabling value exchange from one party to another. An actionable strategy would meet state law requirements and operate independently of existing platforms likely to end services as a result of regulatory changes or become obsolete as a result of market complexities.

In closing, as the meta verse becomes an ever-present social reality, immersive sound design applied to the results of the sonification is of increasing importance. Audio samples, when called by the Call & Response processing, take on the acoustic properties of the space identified by the performer. However, the system currently offers the performer a stereo listening experience rather than placing the listener inside of a 360-degree listening environment. We are currently looking at making other listening formats, namely 5.1 and 7.1.4 available to the end user.

The work continues.

Progress Report

Carolyn Malachi — Thu, 10 Feb 2022 16:24:47 +0000

INTRODUCTION
The Call & Response ™ system leverages blockchain and digital signal processing technologies to address for virtual environment performers two key problems: silenced audiences and slow, inadequate compensation. The system absorbs audience response data, delivers digital asset payments to performers, and sends to performers sonified audience data which allows them to hear audiences as they might experience them in a physical performance venue. Importantly, Call & Response ™ preserves the privacy of audience members by using only information from their provided comments to guide what the performer hears. Performers interact with Call & Response via decentralized application or website.

PROJECT UPDATE

We intended on a buildout of the Call & Response ™ (CR) system as described in our 2021 US Patent Application 17/451,243. This initial design calls for the creation of a new smart contract, one that performs API calls, sorts audience response data into a format readable by MAX / MSP, and triggers the exchange of digital assets from virtual concert sponsor to performer. Conversations with virtual platform performers (artists, DJs, etc.), audio engineers specializing in surface property modeling and rendering, developers in the blockchain space, and Grant for the Web representatives provided insight which refined our concept.

Integrating Interledger Protocol and Web Monetization
Performers on virtual platforms need the ability to receive accurate, timely compensation from audiences which span the globe. In an ideal situation, audiences, sponsors, and other supporters remit payment in the currency of their choice, whether fiat or cryptocurrency, and performers receive payment in their respective currency of choice. Coil, a founding Grant for the Web collaborator, already makes this possible through its use of the open Interledger protocol. Coil Members send micropayments to streamers on monetized sites such as Twitch. With a Coil membership, Twitch Affiliates or Partners in the US can receive USD for each second a Coil member watches their stream. The CR system promises payments based on thresholds set by the inflow of audience responses, but the time-based compensation established by Coil is a reasonable compromise. While Coil does automate payment for Twitch streams, as the CR system requires, streamers could potentially wait 24 hours or more to receive payment. A later iteration of CR will look to address this.

Sonifying Silent Virtual Audiences
With Coil meeting CR’s payment goals, our priority shifted to sonifying silenced virtual audiences. We accomplish this by connecting to a Twitch streamer’s live chat through IRC and saving chat details as they come in. Chat response details are limited to timestamp, username, chat content, and the streamer’s channel name. CR uses public information, thereby preserving the privacy of streamers and viewers. Saved to a readable format, chat responses are passed into a MAX patcher via OSC messages. Responses are mapped to controls that trigger playback of audience audio samples recorded at live music events. Using Spat5, an audio spatialization software package which integrates with MAX, we can position audience audio samples in various sound field locations. We can adjust various parameters like reverb and distance. This allows CR to deliver to the performer an immersive audio experience, one that allows them to hear in a virtual environment the audience as they would sound in a physical performance environment. Currently, if the end user, the performer, wants to change the way they hear the audience, for example, from Jazz club to arena, they would use TouchOSC on the mobile device connected to their in-ear monitors. This is the interface where they select audio sample packs and choose different venue settings which, for our purposes, means the manipulation of spatial sound characteristics.

PROGRESS ON OBJECTIVES

Phase 1: Sonification and Acoustic Modeling buildout
MAX patcher is receiving Twitch stream comments (GitHub repo available upon request), executing audience audio sample playback, and applying spatial sound characteristics based on Spat5 settings. We are committed to feeding CR with high-fidelity audio samples, recorded in real performance environments. To accomplish this, we have acquired field production equipment and studio outboard equipment to record, edit, and mix sound that delivers to the performer the authentic live-performance feeling. This includes sounds of live-concert audiences, room tone, and other ambient sounds. We purchased production and post-production equipment during the pre-grant period. However, emerging COVID-19 variants complicated our production schedule by creating unsafe conditions and canceling live music events. As COVID is expected to abate as the weather changes, a return to field production now seems appropriate. We expect to support four advanced Audio Production students from the Cathy Hughes School of Communications at Howard University as interns who will work with us during the summer months (May - July) as field recordists and post-production sound mixers.

Phase 2: Smart Contract creation with Interledger Protocol integration; Start project personnel + Phase 3: Combine interledger; Progress report due
We are using Coil integrated with Twitch. Project personnel include MaSheva King, Research Lead; Michael Fioretti, Software Engineer; Carmelita Group, Consultant; and several music professionals.

Phase 4 calls for beta testing with 100 virtual performers and content creators, as well as the beginning of our public relations / mission-driven awareness campaign. Our strategy is to do two public tests via Twitch live stream: May 7 with canned audio samples and July 9 with high-fidelity audience sound recordings fully implemented. We have a soft commitment from an academician whose research areas are music, audio, and DSP. He would join the team during the summer months to implement enhancements to the sonification side of the project. The target date range for public testing is August 1 to September 1. Therefore, we are submitting a No-Cost Extension Request for a revised project end date of January 31, 2023 which would provide valuable time to complete Phases 5 and 6.

COMMUNICATION & MARKETING

When we announced the Grant for the Web sponsorship, we received several messages from artists, music producers, and engineers intrigued by the idea of Call & Response ™ . Those conversations included discussions about Interledger Protocol and Web Monetization technologies. We now understand that our demonstration of CR will help other artists appreciate the system and the potential of Interledger Protocol and Web Monetization. Because new music releases generate significant social media interest, our two Twitch live streams, which will serve as public tests of CR, are directly tied to Carolyn Malachi’s single release (May) and album release (July). We plan for musicians who participated in the recording of the album to participate in the marketing and promotion of Call & Response project by hosting their own Twitch live streams to test the project.

Beyond organic reach and mission-driven campaign execution with the Carmelita Group, we would like to bring on the right PR firm to amplify our message of access and equity amongst the community of professional creators working in virtual spaces. Our present challenge is to find a winning team that will work within our budget from July through November.

Previously, we mentioned producing high-fidelity audio sound recordings. These recordings are essential to providing the end user with a lifelike, immersive sonic experience. We intend to quietly engage members of the music community after their live performances through demonstrations, one-on-one discussions, and NFT giveaways to increase music community interest.

WHAT COMMUNITY SUPPORT WOULD BENEFIT YOUR PROJECT?

We need help with UX design. The CR team could benefit from the expertise of technical professionals to assemble all CR elements in one place. Twitch, through Coil, facilitates our asset exchange, but we extract audience response data from Twitch, import it into MAX to execute sonification, and then give the end user controls via TouchOSC.

Finally, we welcome musicians, DJs, dancers, and other performers who are part of the community to help us test CR before we open testing to the public. Finally, we would absolutely love for the community to spread the word once the project is ready for public access.