Final Report

OVERVIEW

The Call & Response system (U.S. Patent Publication No. US-2022-0122570-A1) addresses two problems for individuals (referred to hereafter as “performers”) operating in the virtual performance space: payment friction and inaudible audiences. A matter made urgent during the height of the COVID-19 pandemic for music professionals who found themselves involuntarily out of normal work, also proved consequential for recording engineers, producers, songwriters, music instructors, and other professionals whose work shifted from in-person and relationship dependent to online and anonymously associated. Call & Response sought to and digital signal processing technologies to achieve its goal while preserving the privacy of audience members using only information from their live-virtual event feedback (comments, emoji, etc.) to guide what the performer or practitioner hears and how much digital currency that person or entity receives.

MAX / MSP patch converting incoming live Twitch audience responses into audible audience feedback

RESEARCH

When designing Call & Response, the question of whom the system needed to reach and if they would be likely to adopt the system once complete was raised. Women, young workers, and racial and ethnic minorities were among the major demographic groups experiencing the greatest increases in unemployment during the pandemic. Studies show that prior to the pandemic, women who worked in the music industry were already behind the economic curve before the pandemic with 90% working multiple jobs to earn a livable wage. Also, among the music industry’s studied population of producers and engineers, reportedly only 3% are women, a figure that is oft refuted by the population itself. Women working in the music industry appeared poised to benefit the most from the Call & Response system, especially given their daily interaction with technology. Neither of the available quantitative analyses included information about open payments or whether this demographic had used blockchain-based technologies to support themselves economically.

To acquire this information, we conducted qualitative research, traveling to Los Angeles, New York, and Nashville to interview some of the industry’s top producers and engineers. Our interviews centered around their pandemic-dictated changes to work and their comfort with new payments technologies. More than 75% of respondents reported hearing of such technologies but perceptions of illegality and complicated process dissuaded them. These interviews were critical to developing an understanding of how the technology should be presented to the target demographic. Interview subjects gave the sense that this technology is for other people, and not for them. How could this technology serve this group if they were not interested in it or were unable to identify with people they perceive to be the main users?

To hear excerpts of our qualitative research, click here.

Stated perceptions of users of this technology centered around ideas that current users adopt this technology to participate in illegal activity, or that the technology was too advanced, or that pain was an inevitability of growing to learn the technology, albeit full of promise. We understood that education, at the end of project development, would be a critical support to the marketing efforts being designed. We set aside 20% of our budget, $20,000, in a BlockFi interest bearing account, to prepare to support that effort.

DEVELOPMENT

Call & Response (patent pending number) combines two systems, both governed by an API call that extracts real-time data sets containing the verbal (text) responses from viewers of live streamed digital content. The incoming data sets trigger the playback of audience audio samples designed to model the surface properties and acoustic parameters of known performance environments, allowing the performer to hear once silenced virtual audiences as they would in their venue of choice. Simultaneously, the incoming information, according to the original Call & Response design, maps to certain parameters in a smart contract. The information, depending on the range in which it falls, triggers the release of predetermined amounts of a digital asset from one party (virtual event sponsor or promoter) to another (virtual performer / producer / instructor, etc.). The open payments layer of the system, the smart contract, would enable concert supporters to provide compensation in any currency while the performer would be paid out in the currency of their choice.

Strategy

Understanding the existing hesitancy among our focus population, we sought to leverage Twitch’s relationship with Coil. Doing so would allow us to model the web monetization feature of the Call & Response system using a well-known platform which would automatically, through its Coil integration, facilitate the delivery of digital assets (timely micro payments) to the virtual performer. We collected data (commentary, time of commentary, and username) from the live feed of the video. These bits of information were organized into a spreadsheet which fed the inputs of a sonification patch built in MAX / MSP. Inflowing data automated the playback of audience audio samples. Data thresholds determined which audio samples were played and made audible for the performer.

Expenses

With a process outlined, marketing professionals were hired for a five-month contract with the goal of developing a roll-out plan which included product testing among the designated target population. Based in Charlotte, NC, the firm is founded and operated by Black women. In addition to market research conducted to understand our target audience, we hired student interns to work on sound design for the project. Their work was supported by the purchase of computers, software, physical signal (outboard) processing gear, and other audio production equipment. This project also hired management staff and two developers. The first 80% of the grant payment covered the aforementioned expenses. With the expectation of moving into project testing at the end of 2022, the remaining 20% payment was held in a interest account with the goal of earning interest from the staking of USDC. Those funds remain in BlockFi’s custody through its ongoing Bankruptcy hearing.

CHALLENGES

The Fintech landscape evolved dramatically during the grant period. Changes to government regulation at the state level, custodian bankruptcy, and partner closing impacted our ability to execute the project as planned. On July 13, 2022, the Department of Labor in the State of Maryland updated its financial regulations governing money transmitters to include individuals or organizations facilitating virtual currency exchange and trading services. Call & Response is subject to this regulatory action, as the design is meant to do just that. The license application fee is nominal. Yet, the new law establishes that money transmitters, individuals and organizations engaged in the activity of currency exchange, including virtual currency exchanging and trading services, hold $150,000 of surety bonds. The state’s Fintech Innovation Contact exists to assist with navigating the compliance process, but the bond requirement exists nonetheless. The Interledger Foundation team, understanding of the complicated regulatory issue, suggested creating a model framework for the system, an entirely feasible strategy at the time. Also, an Interledger community member, Dr. Andrew Mangle at Bowie State University provided other suggestions for obtaining the necessary support to move forward with operation.

July 13, 2022: State of Maryland money transmitter regulation includes entities engaging in the exchange of virtual currencies

July 13, 2022: State of Maryland's Money Transmitter License application requires a $150,000 bond for each of the currencies supported by Call & Response.

Without the funding required by the State of Maryland to obtain a Money Transmitter license, we decided to leverage Coil’s products to demonstrate how incoming data from live virtual performance could initiate open payments for performers. We knew Coil as an established entity that could handle the transfer of digital assets while we developed the code to execute the sonification component. Testers of the Call & Response system would sign up for Coil accounts during the testing period to both experience the sonification and relatively real-time payments resulting from the flow of data from their live Twitch sessions. Be that as it may, Coil announced in January that it would sunset its products and development. It would no longer accept sign-ups from new users as of February 2, and it would discontinue its services altogether on March 15. This news derailed our testing plan.

Feb 2, 2023: Coil sunsets its products and prevented new users from signing up, thereby derailing our intended workaround for testing

Finally, with the first grant remittance mostly consumed by research and marketing campaign design, we felt it important to preserve for market testing the remaining grant funds which were paid after the first progress report. BlockFi had a strong reputation for safe storage as well as fair staking. We decided to hold and grow this $20k as USDC in a BlockFi interest account. Since then, then downfall of FTX and Alameda have been well documented, and BlockFi’s bankruptcy case thwarted by delays. Our budget for market testing remains tied up in the BlockFi interest account as the company has since November paused withdrawals.

November 11, 2022: BlockFi pauses user withdrawals as a result of the FTX / Alameda collapse. Education and testing budget still held there.

INSIGHTS

First, an opportunity for community education persists. Perceptions of illegality and technical complication create great apprehension toward open networks, digital assets, and other developments in financial technologies that expand access to financial resources beyond conventional financial services. We are developing a podcast that focuses on our target audience’s pandemic-era economic changes and their feelings toward fintech. It’s an opportunity to mitigate misinformation through community dialogue.

The current design of Call & Response necessitates public education not only on the subject of open payments and network interoperability, it also requires the user to understand visual programming languages and how to operate those platforms. The sonification component of Call & Response is conducted by a MAX/MSP patch. When opened, the patch automatically loads the spreadsheet to which live event data is stored, and that data triggers the playback of audience audio samples. However, programmed differently, with other audio samples, the output of the patch provides for the performer musical elements which could be incorporated in or released as new musical compositions. The performer then has an opportunity to, as they would with other work, generate sustaining income from royalties.

From one perspective, Call & Response in its current state successfully converts incoming data from live virtual events into audible audience responses which are pre-recorded audio samples. Data thresholds govern the timing of sample playback, and performer preferences dictate the acoustic properties applied to samples. From another perspective, the project also successfully provides as a tool for income generation the results of the sonification which an artist can place in the existing digital music market or the NFT market allowing for participation in open payments networks where the performer can set their own market rate and experience transparent payments, which differs from the traditional royalties system. Where Call & Response needs to make up ground is in its strategy for enabling value exchange from one party to another. An actionable strategy would meet state law requirements and operate independently of existing platforms likely to end services as a result of regulatory changes or become obsolete as a result of market complexities.

In closing, as the meta verse becomes an ever-present social reality, immersive sound design applied to the results of the sonification is of increasing importance. Audio samples, when called by the Call & Response processing, take on the acoustic properties of the space identified by the performer. However, the system currently offers the performer a stereo listening experience rather than placing the listener inside of a 360-degree listening environment. We are currently looking at making other listening formats, namely 5.1 and 7.1.4 available to the end user.

The work continues.