Progress Report

INTRODUCTION
The Call & Response ™ system leverages blockchain and digital signal processing technologies to address for virtual environment performers two key problems: silenced audiences and slow, inadequate compensation. The system absorbs audience response data, delivers digital asset payments to performers, and sends to performers sonified audience data which allows them to hear audiences as they might experience them in a physical performance venue. Importantly, Call & Response ™ preserves the privacy of audience members by using only information from their provided comments to guide what the performer hears. Performers interact with Call & Response via decentralized application or website.

PROJECT UPDATE

We intended on a buildout of the Call & Response ™ (CR) system as described in our 2021 US Patent Application 17/451,243. This initial design calls for the creation of a new smart contract, one that performs API calls, sorts audience response data into a format readable by MAX / MSP, and triggers the exchange of digital assets from virtual concert sponsor to performer. Conversations with virtual platform performers (artists, DJs, etc.), audio engineers specializing in surface property modeling and rendering, developers in the blockchain space, and Grant for the Web representatives provided insight which refined our concept.

Integrating Interledger Protocol and Web Monetization
Performers on virtual platforms need the ability to receive accurate, timely compensation from audiences which span the globe. In an ideal situation, audiences, sponsors, and other supporters remit payment in the currency of their choice, whether fiat or cryptocurrency, and performers receive payment in their respective currency of choice. Coil, a founding Grant for the Web collaborator, already makes this possible through its use of the open Interledger protocol. Coil Members send micropayments to streamers on monetized sites such as Twitch. With a Coil membership, Twitch Affiliates or Partners in the US can receive USD for each second a Coil member watches their stream. The CR system promises payments based on thresholds set by the inflow of audience responses, but the time-based compensation established by Coil is a reasonable compromise. While Coil does automate payment for Twitch streams, as the CR system requires, streamers could potentially wait 24 hours or more to receive payment. A later iteration of CR will look to address this.

Sonifying Silent Virtual Audiences
With Coil meeting CR’s payment goals, our priority shifted to sonifying silenced virtual audiences. We accomplish this by connecting to a Twitch streamer’s live chat through IRC and saving chat details as they come in. Chat response details are limited to timestamp, username, chat content, and the streamer’s channel name. CR uses public information, thereby preserving the privacy of streamers and viewers. Saved to a readable format, chat responses are passed into a MAX patcher via OSC messages. Responses are mapped to controls that trigger playback of audience audio samples recorded at live music events. Using Spat5, an audio spatialization software package which integrates with MAX, we can position audience audio samples in various sound field locations. We can adjust various parameters like reverb and distance. This allows CR to deliver to the performer an immersive audio experience, one that allows them to hear in a virtual environment the audience as they would sound in a physical performance environment. Currently, if the end user, the performer, wants to change the way they hear the audience, for example, from Jazz club to arena, they would use TouchOSC on the mobile device connected to their in-ear monitors. This is the interface where they select audio sample packs and choose different venue settings which, for our purposes, means the manipulation of spatial sound characteristics.

PROGRESS ON OBJECTIVES

Phase 1: Sonification and Acoustic Modeling buildout
MAX patcher is receiving Twitch stream comments (GitHub repo available upon request), executing audience audio sample playback, and applying spatial sound characteristics based on Spat5 settings. We are committed to feeding CR with high-fidelity audio samples, recorded in real performance environments. To accomplish this, we have acquired field production equipment and studio outboard equipment to record, edit, and mix sound that delivers to the performer the authentic live-performance feeling. This includes sounds of live-concert audiences, room tone, and other ambient sounds. We purchased production and post-production equipment during the pre-grant period. However, emerging COVID-19 variants complicated our production schedule by creating unsafe conditions and canceling live music events. As COVID is expected to abate as the weather changes, a return to field production now seems appropriate. We expect to support four advanced Audio Production students from the Cathy Hughes School of Communications at Howard University as interns who will work with us during the summer months (May - July) as field recordists and post-production sound mixers.

Phase 2: Smart Contract creation with Interledger Protocol integration; Start project personnel + Phase 3: Combine interledger; Progress report due
We are using Coil integrated with Twitch. Project personnel include MaSheva King, Research Lead; Michael Fioretti, Software Engineer; Carmelita Group, Consultant; and several music professionals.

Phase 4 calls for beta testing with 100 virtual performers and content creators, as well as the beginning of our public relations / mission-driven awareness campaign. Our strategy is to do two public tests via Twitch live stream: May 7 with canned audio samples and July 9 with high-fidelity audience sound recordings fully implemented. We have a soft commitment from an academician whose research areas are music, audio, and DSP. He would join the team during the summer months to implement enhancements to the sonification side of the project. The target date range for public testing is August 1 to September 1. Therefore, we are submitting a No-Cost Extension Request for a revised project end date of January 31, 2023 which would provide valuable time to complete Phases 5 and 6.

COMMUNICATION & MARKETING

When we announced the Grant for the Web sponsorship, we received several messages from artists, music producers, and engineers intrigued by the idea of Call & Response ™ . Those conversations included discussions about Interledger Protocol and Web Monetization technologies. We now understand that our demonstration of CR will help other artists appreciate the system and the potential of Interledger Protocol and Web Monetization. Because new music releases generate significant social media interest, our two Twitch live streams, which will serve as public tests of CR, are directly tied to Carolyn Malachi’s single release (May) and album release (July). We plan for musicians who participated in the recording of the album to participate in the marketing and promotion of Call & Response project by hosting their own Twitch live streams to test the project.

Beyond organic reach and mission-driven campaign execution with the Carmelita Group, we would like to bring on the right PR firm to amplify our message of access and equity amongst the community of professional creators working in virtual spaces. Our present challenge is to find a winning team that will work within our budget from July through November.

Previously, we mentioned producing high-fidelity audio sound recordings. These recordings are essential to providing the end user with a lifelike, immersive sonic experience. We intend to quietly engage members of the music community after their live performances through demonstrations, one-on-one discussions, and NFT giveaways to increase music community interest.

WHAT COMMUNITY SUPPORT WOULD BENEFIT YOUR PROJECT?

We need help with UX design. The CR team could benefit from the expertise of technical professionals to assemble all CR elements in one place. Twitch, through Coil, facilitates our asset exchange, but we extract audience response data from Twitch, import it into MAX to execute sonification, and then give the end user controls via TouchOSC.

Finally, we welcome musicians, DJs, dancers, and other performers who are part of the community to help us test CR before we open testing to the public. Finally, we would absolutely love for the community to spread the word once the project is ready for public access.