2020-12-21: Machine Learning on Browser with TensorFlow

What is TensorFlow and TensorFlow.js

 

TensorFlow is an open-source library developed for machine learning, which is widely used for deep neural networks. The library provides functionalities for developing, training, and testing machine learning models. The TensorFlow Python API provides convenient means of accessing the library where execution happens based on code written in C++.

TensorFlow.js is the JavaScript API of the TensorFlow library, making it possible to add machine learning capabilities to any web application or any Node.js application. Like the Python API, the JavaScript API provides the functionalities for developing, training, and testing machine learning applications.

Furthermore, the JavaScript API allows the models we have developed using Python to be deployed to a web browser with no to minimal changes. In this blog post, we will go through some of TensorFlow's basic concepts shared between different APIs, train a model on the browser, and deploy a simple model on a regular web page.

Basic Concepts of TensorFlow

Backend

In TensorFlow, the backend refers to the execution environment where the operations are executed. In TensorFlow Python API, we can choose between CPU and Graphic Processing Unit (GPU) depending on the availability of Compute Unified Device Architecture (CUDA) accelerated GPU. When we select CPU as the backend, all the tensor operations are carried out by the CPU. Since the functions are mostly matrix operations, GPUs are can perform these operations much faster than CPUs.

Similar to Python API, TensorFlow.js also provides the functionality of using either CPU or GPU as the backend. However, we have several options when accessing the GPU. When running a JavaScript application with Node.js, we can access the CUDA accelerated GPU by selecting the "tensorflow" backed. In the case of a browser application, we can use WebGL to access the GPU of the host machine.

Tensors

Tensors are the data structure or the data unit that is used within the library. The tensors resemble multi-dimensional arrays; however, they have been optimized for tensor operations, which we will discuss in the next section. Further, depending on the backend you use for your code, the Tensors can reside on the main memory or GPU memory. We can create tensors by using the "tf.tensor()" method.

When using TensorFlow python API,  you do not need to worry about garbage collection. Whenever you lose track of a variable, it automatically gets collected by the garbage collector. However, in TensorFlow.js, when we are using the WebGL backend, there's no garbage collector since the data is stored as WebGL Textures. So, suppose our code is going to use WebGL as the backend. In that case, it is mandatory to include disposal operations in our code to prevent any GPU memory leaks.

Also, no issue can arise when we include disposal operations in our non-WebGL environment code. Instead, it can help free up memory faster than the regular garbage collector available in the backend.

Operations

We use TensorFlow operations to manipulate data in tensors. The procedures provided in the JavaScript API ranges from basic arithmetic operations to advanced functions such as Fourier transforms, convolutions, and linear algebraic operations.

Due to the tensors' immutability, each time we run any operation, it will return a new tensor. As mentioned earlier, if we are running on WebGL, we have to dispose the new tensors after when we don't need them. Otherwise, we will run into a memory issue.

Models

In machine learning, models are functions or procedures with stored parameters that transform the input into an output. During training, we optimize the stored parameters so that the model provides the desired form of output. The same concept is applied in TensorFlow and TensorFlow.js.

TensorFlow.js provides two primary means of creating models by using either "tf.Sequential" API or using "tf.model" API. The former is simple and easier to understand since it resembles our perception of a model where inputs pass through layers. Hence we will limit our discussion to the "tf.Sequential".

When creating a model using "tf.Sequential", we can either specify all the layers at once or add layers to the model as we proceed. If we chose the latter, we would have to use "model.add()" function, passing the layer as the parameter. In the former's case, we have to specify all the layers in the model according to the topology of the model. The following examples illustrate the same model being constructed through the two different approaches mentioned.

A Beginner Application

Let's see how we can apply the concepts mentioned in a simple web application. I've included a sample web application that contains the code for an application that converts Fahrenheit to Celsius in the range 0-100 Fahrenheit. In the application, a neural network trained for some Celsius and Fahrenheit readings will perform the conversion.

Beginner application: Fahrenheit-2-Celsius

 

First, make sure you have Node Package Manager (npm) installed on your machine. If not, you can start by installing npm by following the official installation guide. Once you have npm installed, the following command should return the version of your npm installation.

Next, clone the repository into your device and install the dependencies. You can do it by issuing the following commands.

The boilerplate project includes an "index.html" file along with an "index.ts" file that contains application logic. Throughout the discussion, we will use TypeScript for coding, which is a superset of JavaScript. The boilerplate also includes parcel-bundler as the web application bundler, which will bundle our dependencies and compile TypeScript code into JavaScript. Once all the dependencies are installed, you can locally server the web application by running the following command. This will invoke the bundler to deploy the web application through a port on your machine.

If you go through the package.json file in the project, you will see all the necessary dependencies mentioned along with TensorFlow.js. Therefore a successful dependency installation will install TensorFlow.js. Upon successful installation, you can import the TensorFlow.js library by including the following code snippet in your code.

The next step of our web application is to define and build the model. Since Fahrenheit and Celsius's real relationship is linear, we will be using a simple multi-layer perception model. The network will contain a single node for input and another single node for the output. Between the input and output layers, we can have multiple layers of different sizes. For this illustration, let's use a layer with five neurons in between.

Neural network architecture for beginner application

 

We can code the model using the "Sequential" API we discussed earlier as follows. In addition to specifying the structure of the model, we have to specify the loss function and the optimizer to use for the model. We can do it by calling "model.compile()" and passing the loss function and optimizer the model.

During the training phase, the loss is minimized for the examples provided by following the optimization methodology specified in the optimizer. Here we will be using the mean squared error as the loss and the Adam optimizer as the optimizer.

Now we have the model ready for training. But we don't have any training examples. For that, we will generate some training examples by using the actual formula used for Fahrenheit-Celsius conversion. For that, we can use the following code snippet in our application.

As the final step of training the model, we can prepare the model using the examples generated by invoking the "model.fit()" function. We can also specify the number of iterations for the model to train by sending "epochs" as an additional argument to the "fit()" function.

Once our model is trained, we can use the trained model to predict for a given input. In this case, we can use the trained model to convert Fahrenheit temperature readings to Celsius readings. We can do this by calling the "model.predict()" function passing Fahrenheit readings.

Advanced Example

The Trick-o-Research game is an example of a more advanced application of TensorFlow.js developed by NirdsLab. The game uses the video feed from the web camera to determine the gaze position of the user. All the processing required up to gaze position estimation is performed in the browser with the help of TensorFlow.js. 

Gaze interaction game architecture

A gameplay capture. Goal is to navigate blue box avoiding green boxes.

GitHub Link: https://github.com/nirdslab/trick-or-treat.git

Demo Link: https://www.cs.odu.edu/~bhanuka/game.html

-- Bhanuka Mahanama (@mahanama94)