diff --git a/docs/GUIDE.md b/docs/GUIDE.md new file mode 100644 index 0000000..81e0d8c --- /dev/null +++ b/docs/GUIDE.md @@ -0,0 +1,385 @@ +# Add Realtime 3D Avatars to Live Video Streams + +In today's rapidly evolving digital landscape, live-stream video is dominating real-time communication. Users now expect more immersive and customizable streaming options. Content creators are increasingly seeking creative new ways to stream themselves, giving rise to the demand for dynamic 3D avatars that mirror their movements and expressions. + +![](/docs/images/Intro-Demo.gif "Real-Time Avatars with Agora") + +Real-time virtual avatars traditionally required complex motion capture equipment and sophisticated software, often making them inaccessible to everyday users and independent creators. However, this is another area where artificial intelligence is changing the status quo. With advancements in computer vision, it's now possible to run sophisticated AI algorithms on-device that can accurately capture and translate human facial gestures into digital form in real-time. + +In this walkthrough, we'll look at how to integrate 3D virtual avatars into your [Agora](https://www.agora.io) live streams using [MediaPipe](https://ai.google.dev/edge/mediapipe/solutions/guide) and 3D avatars from [ReadyPlayerMe](https://readyplayer.me/). Whether you're looking to enhance audience engagement or just add a fun, creative twist to your app's video calls/live broadcasts, this guide will provide you with the necessary steps to bring 3D virtual personas to life. + +## Prerequisites +- [Node.JS](https://nodejs.org) +- A developer account with [Agora](https://console.agora.io) +- A basic understanding of HTML/CSS/JS +- A basic understanding of ThreeJS +- A basic understanding of Agora - [Web QuickStart](https://medium.com/agora-io/a-simple-approach-to-building-a-group-video-chat-web-app-0b8a6cacfdfd) +- A code editor, I use [VSCode](https://code.visualstudio.com) +- A 3D avatar from [ReadyPlayerMe](https://readyplayer.me/) + +## Agora + MediaPipe project +To keep this guide concise, I assume you understand how to implement the Agora Video SDK into a web app; if you don't, check out my guide on [ Building a Group Video Chat Web App](https://medium.com/agora-io/a-simple-approach-to-building-a-group-video-chat-web-app-0b8a6cacfdfd). + +To get started, download the [demo project](https://github.com/digitallysavvy/agora-mediapipe-readyplayerme). With the code downloaded, navigate to the project folder in the terminal and use `npm` to install the node packages. + +```bash +git clone git@github.com:digitallysavvy/agora-mediapipe-readyplayerme.git +cd agora-mediapipe-readyplayerme +npm i +``` + +## Core Structure (HTML) +Let’s start with the HTML structure in [`index.html`](index.html), at the top of the `` are the "call" UI elements. These include a container for the remote videos, a container for the local user with buttons for muting and unmuting the audio/video, and a button to leave the chat. + +Aside from the call UI, we'll need an overlay screen for users to input the URL to their avatars, and a button to join the channel. + +```HTML + + + + + + + + Agora Live Video Demo + + +
+
+
+ + + +
+ + + + +``` + +## Agora Client and data stores +In [`main.js`](/main.js) we create a new Agora client to use Agora's SDK and use `localMedia` to keep a reference to the audio, video, and canvas tracks and their active state. We'll need `headRotation` and `blendShapes` to store the data we get from MediaPipe's computer vision. + +```javascript +// Create the Agora Client +const client = AgoraRTC.createClient({ + codec: 'vp9', + mode: 'live', + role: 'host' +}) + +const localMedia = { + audio: { + track: null, + isActive: false + }, + video: { + track: null, + isActive: false + }, + canvas: { + track: null, + isActive: false + }, +} + +// Container for the remote streams +let remoteUsers = {} + +// store data from facial landmarks +let headRotation +let blendShapes + +``` + +### DOMContentLoaded and Event Listeners +When the page loads, we'll add listeners for the Agora events, the media controls, and the form submission. With the listeners in place, we're ready to show the overlay form. + +```javascript +// Wait for DOM to load +document.addEventListener('DOMContentLoaded', async () => { + // Add the Agora Event Listeners + addAgoraEventListeners() + // Add listeners to local media buttons + addLocalMediaControlListeners() + // Get the join channel form & handle form submission + const joinform = document.getElementById('join-channel-form') + joinform.addEventListener('submit', handleJoin) + // Show the overlay form + showOverlayForm(true) +}) +``` + +> NOTE: Make sure to add client event listensers before joining the channel, otherwise some events may not get triggered as expected. + +## 3D & Avatar Setup +One of the Prerequisites for this guide is a 3D avatar from ReadyPlayerMe because ReadyPlayerMe provides 3D files that adhere to the name conventions outlined by [Apple's ARKit ARFaceAnchor locations](https://developer.apple.com/documentation/arkit/arfaceanchor/blendshapelocation). These definitions are industry standard and match the output from MediaPipe. + +Getting back to the code, when the user clicks the "Join" button, initialize the ThreeJS scene and append the `` to the `localUserContainer`. + +```javascript +// get the local-user container div +const localUserContainer = document.getElementById('local-user-container') + +// create the scene and append canvas to localUserContainer +const { scene, camera, renderer } = await initScene(localUserContainer) +``` + +Using the newly created scene, load the user's ReadyPlayerMe avatar using the `glbURL`. You'll notice URL parameters are appended to the `glbURL`. This is because blend shapes are not part of the default `.glb` file provided by ReadyPlayerMe. These parameters are part of the [ReadyPlayerMe RESTful API for Avatars](https://docs.readyplayer.me/ready-player-me/api-reference/rest-api/avatars/get-3d-avatars). + +Once the 3D avatar is loaded, we'll traverse its scene graph and create an object with all the nodes. This will give us quick access to the `headMesh`. + +```javascript +// append url parameters to glb url - load ReadyPlayerMe avatar with morphtargets +const rpmMorphTargetsURL = glbURL + '?morphTargets=ARKit&textureAtlas=1024' +let nodes +// Load the GLB with morph targets +const loader = new GLTFLoader() +loader.load(rpmMorphTargetsURL, + async (gltf) => { + const avatar = gltf.scene + // build graph of avatar nodes + nodes = await getGraph(avatar) + const headMesh = nodes['Wolf3D_Avatar'] + // adjust position + avatar.position.y = -1.65 + avatar.position.z = 1 + + // add avatar to scene + scene.add(avatar) +}, +(event) => { + // outout loading details + console.log(event) +}) +``` + +To account for the noticeable delay between the time the scene is initialized and the moment the 3D avatar is loaded. It's good practice to display a loading animation to inform the user the model is loading and remove it once the 3D avatar is added to the scene. + +```javascript +// show a loading animation +const loadingDiv = document.createElement('div') +loadingDiv.classList.add('lds-ripple') +loadingDiv.append(document.createElement('div')) +localUserContainer.append(loadingDiv) + +/* loader.load - success callback */ +loadingDiv.remove() // remove the loading spinner +``` + +## Init video element with Agora +![](/docs/images/Agora-to-video_element.png "Use Agora Video Track in