Python APIs¶

In this section, we will learn the onnc-bench APIs further. We will start from a Pytorch example and use this example to dive into all APIs.

You can find more example in Examples Section

Example¶

import torchvision
from torchvision import datasets, models
import torch
from onnc.bench import login, Project

# Acquire the pretrained Resnet18 Model from TorchVision model zoo
model = models.resnet18(pretrained=True)

# Acquire ImageNet dataset from TorchVision Datasets
imagenet_data = datasets.ImageNet('path/to/imagenet_root/')

# onnc-bench also accept a DataLoader object
data_loader = torch.utils.data.DataLoader(imagenet_data)


login('Your-API-Key')

project = Project(name='experiment-2')
project.add_model(
                  model=model, # feed a Pytorch model object
                  samples=imagenet_data, # feed a Pytorch Dataset object
                  #samples=data_loader, # or you can feed a DataLoader instead
                  model_inputs=[["input", (1, 3, 224, 224), float]]
                  )

project.compile(target='NVDLA-NV-SMALL-DEFAULT')
deployment = project.save('./output')

Project Class¶

Project class represents a project in the remote service. A project in ONNC-SaaS service is used to store, track, debug and manage all experiments and deployments.

from onnc.bench import Project

project = Project(name="experiment-1")

name: str: The name is used to indicate which project a experiment belongs to. If the project name exists, the project will be re-used; Otherwise, ONNC-SaaS will create a new project to accommodate your models.

Project.add_model Method¶

add_model is used to provide a model and its coressponding calibration samples to the compieler.

project.add_model(
    model: Union[object, str, Path],
    samples: Union[object, str, Path, Dataset],
    model_format: ModelFormat = ModelFormat.NON_SPECIFIED,
    sample_format: DatasetFormat = DatasetFormat.NON_SPECIFIED,
    model_inputs: List[Union[List, Tuple, Tensor]] = [],
    model_outputs: List[Union[List, Tuple, Tensor]] = [])

model: Union[object, str, pathlib.Path]: A file or a directory path to a serialized model file, or a neurl network model object. Please check Supported Frameworks for the list of supported formats.

samples: Union[object, str, Path, Dataset]: A file or a directory path to a serialized dataset, or a dataset object. Please check Supported Frameworks for the list of supported formats.

model_format: ModelFormat: ONNC Benchcan identify the format of your model if model_format is set to NON_SPECIFIED (default). You can specify the format by yourself. To list the supported format:

>>> from onnc.bench.core.model import ModelFormat

>>> for format in ModelFormat:
>>>    print(format)

ModelFormat.NON_SPECIFIED
ModelFormat.UNKNOWN
ModelFormat.PT_NN_MODULE
ModelFormat.TF_KERAS_MODEL
ModelFormat.KERAS_MODEL
ModelFormat.H5
ModelFormat.SAVED_MODEL
ModelFormat.PB
ModelFormat.PTH
ModelFormat.ONNX
...

sample_format: DatasetFormat: ONNC Bench can identify the format of your calibration dataset if sample_format is set to NON_SPECIFIED (default). You can specify the format by yourself. To list the supported format:

>>> from onnc.bench.core.dataset import DatasetFormat

>>> for format in DatasetFormat:
>>>    print(format)

DatasetFormat.NON_SPECIFIED
DatasetFormat.UNKNOWN
DatasetFormat.NPY
DatasetFormat.NPYDIR
DatasetFormat.NPZ
DatasetFormat.NDARRAY
DatasetFormat.NP_MEMMAP
DatasetFormat.NPZ_OBJECT
DatasetFormat.PB
DatasetFormat.ONNC_DATASET
DatasetFormat.TORCH_DATASET
DatasetFormat.TORCH_DATALOADER
DatasetFormat.KERAS_DATASET
DatasetFormat.TFDS_PREFETCH
...

model_inputs: Tensor: ONNC Bench can identify the inputs of the model if model_inputs is set to [] (default). You can specify the inputs by yourself.

Due to the nature of Pytorch, we have to manually specify model_inputs for Pytorch models

...

# There are two ways to specify signature of input tensors
# Method 1: List[str, Tuple[int], _type]
input_1 = ["input1", (1, 3, 224, 224), float]

# Method 1: Tensor object
from onnc.bench.core.model.model import Tensor
input_1 = Tensor('input_1', (1, 3, 224, 224), ModelDataType.FP32)

project.add_model(
                  model=model, # feed a Pytorch model object
                  samples=imagenet_data, # feed a Pytorch Dataset object
                  #samples=data_loader, # or you can feed a DataLoader instead
                  model_inputs=[input_1]
                  )

...

Definition of Tensor:

from onnc.bench.core.model.model import Tensor
from onnc.bench.core.model import ModelDataType

# The definition of a Tensor class
@dataclass
class Tensor:
    name: str
    shape: Union[None, Tuple[int, ...]] = None
    dtype: Union[None, ModelDataType] = None

# Example of an Tensor object

tensor = Tensor('input_1', (1, 3, 224, 224), ModelDataType.FP32)

model_outputs: Tensor: ONNC Bench can identify the outputs of the model if model_inputs is set to [] (default). You can specify the outputs by yourself.

Project.compile Method¶

compile is used to trigger and start the compilation.

target: str: The name of the supported SoC device, for example, CMSIS-NN-DEFAULT or NVDLA-NV-SMALL-DEFAULT. Please check Supported Devices for the list of supported devices/backends.

Project.save Method¶

save saves the compiled artifacts for inference on device.

path: Union[str, Path]: The path to store the output files

return : an Deployment object

- output/
    - report.json
    - build
        - cortexm_runtime.cpp  
        - cortexm_runtime.h
        - inference.cpp
        - inference.h
        - main.cpp
        - ... (more)

- output/
    - report.json
    - build
        - model.nvdla

Deployment Class¶

Deployment class consolidates the compilation report, logs and a list of artifacts.

report: Dict: A report contains 2 fields:

sram_size: RAM size (in bytes) the model consumes.
flash_size: ROM size (in bytes) the model requires.

logs: List: logs contains logs during compilation. It is usful for debugging a compilation failure.

compiled_files: List: compiled_files is a list of file names generated by TinyONNC.