OpenVino Backend Examples

Prerequisites

In order to use OpenVino to run your model, you need to install OpenVino package, currently we support version 2022.1.0.

pip install openvino==2022.1.0

A Tensorflow Example

import tensorflow as tf
import numpy as np

from onnc.bench import login, Project, Deployment
from onnc.forest.core.runtime import Runtime
from onnc.forest import OpenvinoOptions, Options

# Step 1: Prepare model and dataset

## 1.1 Load a pretrained MobileNet
model = tf.keras.applications.MobileNet(weights="imagenet")

## 1.2 Train or finetune the pretrained model to fit your task
## ... 
## YOUR TRAINING CODE
## ...

# Step 2: Compile and optimize the given model

## 2.1 Log in OASIS
login("YOUR_EMAIL", "YOUR_PASSWD")

## 2.2 Upload and compile the model
project = Project('project-1')
project.add_model(model)
project.compile(target='INTEL-OPENVINO-CPU-FP32')

## 2.3 Download and save the compiled model
deployment = project.save('output/')

# Step 3: Run the optimzed model

## 3.1 Locate the compiled model and loadable
deployment = Deployment('output/')
loadable = deployment.loadable

## 3.2 Create an Options object
options = Options(loadable=loadable)

## 3.3 Create a Runtime object and load the model
runtime = Runtime()
runtime.launch(options)
runtime.load(options)

## 3.4 Bind inputs and outputs
runtime.bind_all_inputs()
runtime.bind_all_outputs()

## 3.5 Materialize loadable
runtime.materialize()

## 3.6 Prepare and preprocess input images
## img = your_preprocessor('path_to_image')
img = np.random.randn(1, 224, 224, 3)

## 3.7 Infer and get result
runtime.write([img])
runtime.run()
res = runtime.read()

## 3.8 Post-process the infered results
## prediction = your_postprocessor(res)
prediction = np.argmax(res)
print(prediction)

A Pytorch Example

The procedure for compiling and running a Pytorch model is almost the same as doing it with Tensorflow, except that we have to specifiy input information of the model.

Due to the nature of dynamic graph in Pytorch, the shapes of the input tensors of a model are unknown until samples are provided. Therefore, we have to specify the input shapes. In order to doing so, just use argument model_inputs when we add a model. For example,

...
project.add_model(model, model_inputs=[("input", [1, 3, 224, 224], float)])
...

Below is a full example for compiling and running a Pytorch MobileNet model.

import torchvision
import numpy as np

from onnc.bench import login, Project, Deployment
from onnc.forest.core.runtime import Runtime
from onnc.forest import OpenvinoOptions, Options

# Step 1: Prepare model and dataset

## 1.1 Load a pretrained MobileNet
model = torchvision.models.mobilenet_v2(pretrained=True)

## 1.2 Train or finetune the pretrained model to fit your task
## ... 
## YOUR TRAINING CODE
## ...

# Step 2: Compile and optimize the given model

## 2.1 Log in OASIS
login("YOUR_EMAIL", "YOUR_PASSWD")

## 2.2 Upload and compile the model
project = Project('project-1')
project.add_model(model, model_inputs=[("input", [1, 3, 224, 224], float)])
project.compile(target='INTEL-OPENVINO-CPU-FP32')

## 2.3 Download and save the compiled model
deployment = project.save('output/')

# Step 3: Run the optimzed model

## 3.1 Locate the compiled model and loadable
deployment = Deployment('output/')
loadable = deployment.loadable

## 3.2 Create an OpenvinoOptions object
options = OpenvinoOptions(loadable=loadable)

## 3.3 Create a Runtime object and load the model
runtime = Runtime()
runtime.launch(options)
runtime.load(options)

## 3.4 Bind inputs and outputs
runtime.bind_all_inputs()
runtime.bind_all_outputs()

## 3.5 Materialize loadable
runtime.materialize()

## 3.6 Prepare and preprocess input images
## img = your_preprocessor('path_to_image')
img = np.random.randn(1, 3, 224, 224)

## 3.7 Infer and get result
runtime.write([img])
runtime.run()
res = runtime.read()

## 3.8 Post-process the infered results
## prediction = your_postprocessor(res)
prediction = np.argmax(res)
print(prediction)