In recent years, the world has seen many major breakthroughs in the field of Machine Learning and so did the libraries used to write these programs. From NumPy to Tensorflow.js, each one faster and equipped with the best available technology of time than the previous one. One such library that we are going to talk about today is ml5.js. ml5 is a JavaScript-based machine learning library built on top of Tensorflow.js which aims to make machine learning approachable for a broad audience of artists, creative coders, and students. The official website of ml5.js states “A neighbourly approach to creating and approaching artificial intelligence in browser”
I recently started writing Deep Learning apps for browsers and can tell you this is pretty much fun and easy when you want to showcase your work to the general public or create an MVP in a hackathon with a cool dashboard and good-looking UI elements. I started programming on the web mainly because of the fact one can easily create an extensive user experience using just CSS and HTML5. Additionally, a web app can be run by any device which has a web browser on it such as a smartphone, desktop, or even a smart tv.
Today we are going to look at how you can create a web app using React.js which can estimate human pose with a very high degree of precision using our neighborly library ml5.js. This example is more focused toward React developers who are looking to use Machine Learning in their upcoming projects. I have used TypeScript, but this code can be easily adapted for JSX.
In the following tutorial you’ll learn:
- How to write a simple React app using Typescript?
- How to use a webcam and canvas in a react app?
- How to use ml5.js in React to find human pose?
- How to draw a skeleton on your image using the output?
To follow along with the tutorial, you should have:
- A basic understanding of React and TypeScript
- A newer version of Node.js installed on your system
For those of you who want to heads up and want to see the code right away here’s the link to GitHub repo: https://github.com/jha-prateek/react_posenet
Creating React App
Open Terminal or Command Prompt and run
| npx create-react-app react-ml5 –template typescript |
This will create a new directory/folder react-ml5 in the current working directory and install all the required dependencies for running and deploying the react app. You will get three subfolders namely node_modules, public, src along with other files.
Importing ml5 library from CDN
Now we will import the ml5 library in our project. To do this we have to go this https://learn.ml5js.org/docs/#/ and get the CDN link.
| <script src=”https://unpkg.com/ml5@0.5.0/dist/ml5.min.js”></script> |
Copy this link and paste it inside your index.html (./react-ml5/public/index.html) between head tags.
React component for Pose estimation
Create a new TypeScript file Pose.tsx component inside components directory (./react-ml5/src/components/Pose.tsx). This is the file where you’ll write the code for running the model and displaying the results.
Declaring ml5 globally
First, you’ll declare the type for using ml5 as a global variable in our code because ml5 is developed in vanilla JavaScript, so it doesn’t have any types defined already. This is important in TypeScript because it’s a strictly typed language.
| declare var ml5: any; |
All types in TypeScript are subtypes of a single top type called the ‘any’ type. As we are unaware of the return type of ml5 so we have declared it as ‘any’ type. Now we can use it freely without bothering the compiler.
Defining DOM elements for Webcam
Let’s define the video element and canvas element for capturing and displaying the live video feed from the webcam. Though the video element itself can be used to display the feed from the webcam, we also need to draw the skeleton and key points so we will be using canvas to draw video and the drawables on top of the canvas.
You’ll also need to declare refs for the same tags. In React refs are the variables that can be used to access the DOM Elements.
Define these class variables
| private webCam: React.RefObject<HTMLVideoElement>; private camCanvas: React.RefObject<HTMLCanvasElement>; |
Initialize these variables inside the class constructor
| this.webCam = React.createRef(); this.camCanvas = React.createRef(); |
Add this to inside the return of render method
| public render() { const camStyle: React.CSSProperties = { display: ‘none’ } return ( <div className=”container”> <canvas ref={this.camCanvas} width=600 height=400 /> <video playsInline ref={this.webCam} width=width=600 height=400 style={camStyle} /> </div> ); } |
Getting the live feed from Client’s webcam
Now let’s query the media devices on the client’s browser and get the live video stream from the webcam. You’ll perform this process inside one of the React’s Lifecycle method componentDidMount(). This Lifecycle method is executed after the first render only on the client-side.
| componentDidMount() { if (!navigator.mediaDevices || !navigator.mediaDevices.getUserMedia) { throw new Error( ‘Browser API navigator.mediaDevices.getUserMedia not available’); } navigator.mediaDevices .getUserMedia({ ‘audio’: false, ‘video’: { facingMode: ‘user’, width: 600, height: 400, frameRate: 30, // Reduce this if there’s a stuttering in feed }, }).then(res => { if (res != null) { this.webCam.current!.srcObject = res; this.webCam.current?.play(); this.initializeModel(); } }); } |
Once you get the result, assign this to the srcObject of the video ref webCam and then play the video. Now you got the live video from webCam.
Note: If you want to check the feed assign set {display: ‘block’} in camStyle object.
Download the pre-trained model in Client’s device
Now we’ll call initializeModel() method to load the ml5 PoseNet model into memory. But, first we need to declare a variable poseNet and poses of type ‘any’ so that we can use it globally just like the ml5 variable. poseNet stores models whereas poses store results.
| private poseNet: any; private poses: any; |
Then we initialize it with the PoseNet model. ml5 PoseNet has a ‘pose’ event listener which returns the result whenever the model detects any pose inside the input image. So, we’ll call the ‘on’ method to set up the ‘pose’ event and get our result inside the callback.
| initializeModel() { this.poseNet = ml5.poseNet(this.webCam.current, () => { console.log(“Model Initilaized”); this.poseNet.on(‘pose’, (result: any) => { this.poses = result; }); }); } |
Once this is done then we store the results in poses to use this variable to find the pose or the coordinates of different body parts.
Creating canvas
Now we need to display the video on canvas as well as draw the results. First, we need to obtain a context object for canvas using getContext() method through camCanvas ref. Using the context object, we can draw 2d shapes on the canvas.
Define a context variable and initialize it with getContext() inside componentDidMount().
| private ctx: CanvasRenderingContext2D; |
Inside componentDidMount()
| this.ctx = this.camCanvas.current.getContext(‘2d’); |
Drawing video and pose on canvas
Now we’ll define a method drawCameraIntoCanvas() which will be used to loop all the animations such as video feed and drawables on canvas. We will also define two more methods drawSkeleton() to draw a line segment between adjacent parts of the body and drawKeypoints() to draw the joints on top of your body which will be called inside the drawCameraIntoCanvas().
| drawKeypoints() { for (let i = 0; i < this.poses.length; i++) { for (let j = 0; j < this.poses[i].pose.keypoints.length; j++) { let keypoint = this.poses[i].pose.keypoints[j]; if (keypoint.score > 0.2) { this.ctx.fillStyle = “#c82124”; this.ctx.beginPath(); this.ctx.arc(keypoint.position.x, keypoint.position.y, 5, 0, 2 * Math.PI); this.ctx.fill(); } } } } drawSkeleton() { for (let i = 0; i < this.poses.length; i++) { for (let j = 0; j < this.poses[i].skeleton.length; j++) { let partA = this.poses[i].skeleton[j][0]; let partB = this.poses[i].skeleton[j][1]; this.ctx.beginPath(); this.ctx.moveTo(partA.position.x, partA.position.y); this.ctx.lineTo(partB.position.x, partB.position.y); this.ctx.stroke(); } } } |
| drawCameraIntoCanvas() { if (this.modelLoaded) { this.ctx.drawImage(this.webCam.current, 0, 0, 600, 400) } if (this.poses !== undefined) { this.drawKeypoints(); this.drawSkeleton(); } requestAnimationFrame(this.drawCameraIntoCanvas.bind(this)); } |
First, the drawCameraIntoCanvas() method is drawing the webCam feed into canvas element when the model is loaded, after that it checks if the model has returned any result. If it finds any then it draws key points as well as the skeleton of the human body on top of video feed. We used something known as requestAnimationFrame() method to loop this entire section and continuously check for pose and draw results on top of the canvas. requestAnimationFrame() method tells the browser that you wish to perform an animation and requests that the browser calls a specified function to update an animation before the next repaint which is drawCameraIntoCanvas() over here.
With this our Pose.tsx component is ready to get imported inside App.tsx.
| import Pose from ‘./components/Pose’; |
And then call the component inside the main body of the react app.
| <div className=”app-body”> <Pose /> </div> |
Here’s the link to full code for Pose.tsx: https://github.com/jha-prateek/react_posenet/blob/master/src/components/Pose.tsx
Feel free to explore and use this code to adapt your needs. Also, if you want to contribute to building more React and ml5 examples with me just create a pull request.
