Perform error checking of ONNX output and TensorFlow output. Verify that the error of all outputs, one operation at a time, is below a certain threshold. Automatically determines before and after which OPs the tool's automatic conversion of the model failed. Know where dimensional compression, dimensional expansion, and dimensional transposition by Reshape and Traspose are failing. Once you have identified the problem area, you can refer to the tutorial on Parameter replacement to modify the tool's behavior.

After many upgrades, the need for JSON parameter correction has become much less common, but there are still some edge cases where JSON correction is required. If the PC has sufficient free space in its RAM, onnx2tf will convert the model while carefully performing accuracy checks on all OPs. Thus, at the cost of successful model conversion, the conversion speed is a little slower. If the amount of RAM required for the accuracy check is expected to exceed 80% of the total available RAM capacity of the entire PC, the conversion operation will be performed without an accuracy check. Therefore, if the accuracy of the converted model is found to be significantly degraded, the accuracy may be automatically corrected by re-conversion on a PC with a large amount of RAM. For example, my PC has 128GB of RAM, but the StableDiffusion v1.5 model is too complex in its structure and consumed about 180GB of RAM in total with 50GB of SWAP space.

-ois an option to overwrite the input OP to a static size if it has undefined dimensions. -cotof option checks the accuracy of all OPs one by one. -cotoa is the error value of the threshold for determining an accuracy error. If there are undefined dimensions in the input OP, it is better to fix them to the static geometry to improve the accuracy of the accuracy measurement.

Also, you can use the -cind option to specify custom input for -cotof, instead of using the default dummy input. Otherwise, all input values will be set to 1. For more information about the -cind option, please refer to here.

The -cotof option only compares the original ONNX and converted TensorFlow (Keras) models at Float32 precision, not at Float16 or INT8 precision.

onnx2tf -i mobilenetv2-12.onnx -ois input:1,3,224,224 -cotof -cotoa 1e-1

or

onnx2tf -i mobilenetv2-12.onnx -b 1 -cotof -cotoa 1e-1

or

onnx2tf -i mobilenetv2-12.onnx -cotof -cotoa 1e-1 -cind "input" "/your/path/x.npy"

If you want to match tflite's input/output OP names and the order of input/output OPs with ONNX, you can use the interpreter.get_signature_runner() to infer this after using the -coion / --copy_onnx_input_output_names_to_tflite option to output tflite file. See: #228

onnx2tf automatically compares the final input/output shapes of ONNX and the generated TFLite and tries to automatically correct the input/output order as much as possible if there is a difference. However, if INT8 quantization is used and there are multiple inputs and outputs with the same shape, automatic correction may fail. This is because TFLiteConverter shuffles the input-output order by itself only when INT8 quantization is performed.

import torch
import onnxruntime
import numpy as np
import onnx2tf
import tensorflow as tf
from tensorflow.lite.python import interpreter as tflite_interpreter

class Model(torch.nn.Module):
    def forward(self, x, y):
        return {
            "add": x   y,
            "sub": x - y,
        }

# Let's double check what PyTorch gives us
model = Model()
pytorch_output = model.forward(10, 2)
print("[PyTorch] Model Predictions:", pytorch_output)

# First, export the above model to ONNX
torch.onnx.export(
    Model(),
    {"x": 10, "y": 2},
    "model.onnx",
    opset_version=16,
    input_names=["x", "y"],
    output_names=["add", "sub"],
)

# And check its output
session = onnxruntime.InferenceSession("model.onnx")
onnx_output = session.run(["add", "sub"], {"x": np.array(10), "y": np.array(2)})
print("[ONNX] Model Outputs:", [o.name for o in session.get_outputs()])
print("[ONNX] Model Predictions:", onnx_output)

# Now, let's convert the ONNX model to TF
onnx2tf.convert(
    input_onnx_file_path="model.onnx",
    output_folder_path="model.tf",
    copy_onnx_input_output_names_to_tflite=True,
    non_verbose=True,
)

# Now, test the newer TFLite model
interpreter = tf.lite.Interpreter(model_path="model.tf/model_float32.tflite")
tf_lite_model = interpreter.get_signature_runner()
inputs = {
  'x': np.asarray([10], dtype=np.int64),
  'y': np.asarray([2], dtype=np.int64),
}
tf_lite_output = tf_lite_model(**inputs)
print("[TFLite] Model Predictions:", tf_lite_output)

[PyTorch] Model Predictions:
  {
    'add': 12,
    'sub': 8
  }
[ONNX] Model Outputs:
  [
    'add',
    'sub'
  ]
[ONNX] Model Predictions:
  [
    array(12, dtype=int64),
    array(8, dtype=int64)
  ]
[TFLite] Model Predictions:
  {
    'add': array([12]),
    'sub': array([8])
  }

If you do not like tflite input/output names such as serving_default_*:0 or StatefulPartitionedCall:0, you can rewrite them using the following tools and procedures. It can be rewritten from any name to any name, so it does not have to be serving_default_*:0 or StatefulPartitionedCall:0.

https://github.com/PINTO0309/tflite-input-output-rewriter

# Install custom flatc
wget https://github.com/PINTO0309/onnx2tf/releases/download/1.7.3/flatc.tar.gz \
&& tar -zxvf flatc.tar.gz \
&& sudo chmod  x flatc \
&& sudo mv flatc /usr/bin/ \
&& rm flatc.tar.gz

# Path check
which flatc
/usr/bin/flatc

# Install tfliteiorewriter
pip install -U tfliteiorewriter

• Before

  

  tfliteiorewriter \
  -i xxxx.tflite \
  -r serving_default_input_1:0 aaa \
  -r StatefulPartitionedCall:0 bbb

  

• After

  

If you want to embed label maps, quantization parameters, descriptions, etc. into your tflite file, you can refer to the official tutorial and try it yourself. For now, this tool does not plan to implement the ability to append metadata, as I do not want to write byte arrays to the tflite file that are not essential to its operation.

• Adding metadata to TensorFlow Lite models

  https://www.tensorflow.org/lite/models/convert/metadata

It is a matter of model structure. The activation function (SiLU/Swish), kernel size and stride for Pooling, and kernel size and stride for Conv should be completely revised. See: #269

If you want to see the difference in quantization error between SiLU and ReLU, please check this Gist by @motokimura who helped us in our research. Thanks Motoki!

Gist: Quantization error simulation of SiLU (Swish) activation

The accuracy error rates after quantization for different activation functions are shown in the figure below. The graph plots the distribution of absolute error, so a position with a higher value on the horizontal axis indicates a larger error. The vertical axis is the number of samples. SiLU (Swish) produces catastrophic errors after INT8 quantization.

• e.g. YOLOX-Nano

  • https://github.com/motokimura/yolox-ti-lite_tflite

  • https://github.com/TexasInstruments/edgeai-yolox

    Before	After
    Swish/SiLU	ReLU
    DepthwiseConv2D	Conv2D
    MaxPool, kernel_size=5x5,9x9,13x13	MaxPool, kernel_size=3x3

    ### Float32 - YOLOX-Nano
    (1, 52, 52, 85)
    array([[[
        [ 0.971787,  0.811184,  0.550566, ..., -5.962632, -7.403673, -6.735206],
        [ 0.858804,  1.351296,  1.231673, ..., -6.479690, -8.277064, -7.664936],
        [ 0.214827,  1.035119,  1.458006, ..., -6.291425, -8.229385, -7.761562],
            ...,
        [ 0.450116,  1.391900,  1.533354, ..., -5.672194, -7.121591, -6.880231],
        [ 0.593133,  2.112723,  0.968755, ..., -6.150078, -7.370633, -6.874294],
        [ 0.088263,  1.985220,  0.619998, ..., -5.507928, -6.914980, -6.234259]]]]),
    
    ### INT8 - YOLOX-Nano
    (1, 52, 52, 85)
    array([[[
        [ 0.941908,  0.770652,  0.513768, ..., -5.993958, -7.449634, -6.850238],
        [ 0.856280,  1.284420,  1.198792, ..., -6.507727, -8.391542, -7.792146],
        [ 0.256884,  0.941908,  1.455676, ..., -6.336471, -8.305914, -7.877774],
            ...,
        [ 0.342512,  1.370048,  1.541304, ..., -5.737075, -7.192750, -7.107122],
        [ 0.513768,  2.226327,  1.027536, ..., -6.165215, -7.449634, -7.021494],
        [ 0.085628,  2.055072,  0.685024, ..., -5.480191, -7.021494, -6.422099]]]]),
    

• Other recommended replacement OP

  Before	After
  HardSwish	ReLU
  ReLU6 Paper: A Quantization-Friendly Separable Convolution for MobileNets https://arxiv.org/pdf/1803.08607.pdf	ReLU

• Quantization range collapse due to non-zero constant padding

  If padding is performed with a constant other than zero, the padding value may destroy the quantization range of the input tensor. For example, the pattern is shown in the figure below. The MaxPool2D is done after padding the 4 sides of the input tensor with the minimum value of Float32. It seems that if INT8 quantization is performed with this structure, the quantization range is determined by MaxPool2D during quantization, including the values padded to the tensor. See: #444

  Therefore, the following two similar examples are equally likely to result in divergent output values for the model after INT8 quantization, with all output values being Nan or zero.

  1. Pattern with fixed value -255.0 padded on 4 sides of tensor

  2. Pattern with fixed value -128.0 padded on 4 sides of tensor

Calibration data (.npy) for INT8 quantization (-cind) is generated as follows. This is a sample when the data used for training is image data. See: #222

https://www.tensorflow.org/lite/performance/post_training_quantization

import cv2
import glob
import numpy as np

# Not used during data generation ################################
# You will need to do the calculations yourself using the test data
MEAN = np.asarray([[[[0.485, 0.456, 0.406]]]], dtype=np.float32) # [1,1,1,3]
STD = np.asarray([[[[0.229, 0.224, 0.225]]]], dtype=np.float32) # [1,1,1,3]
# Not used during data generation ################################

files = glob.glob("data/*.png")
img_datas = []
for idx, file in enumerate(files):
    bgr_img = cv2.imread(file)
    rgb_img = cv2.cvtColor(bgr_img, cv2.COLOR_BGR2RGB)
    resized_img = cv2.resize(rgb_img, dsize=(200,112))
    extend_batch_size_img = resized_img[np.newaxis, :]
    normalized_img = extend_batch_size_img / 255.0 # 0.0 - 1.0
    print(
        f'{str(idx 1).zfill(2)}. extend_batch_size_img.shape: {extend_batch_size_img.shape}'
    ) # [1,112,200,3]
    img_datas.append(extend_batch_size_img)
calib_datas = np.vstack(img_datas)
print(f'calib_datas.shape: {calib_datas.shape}') # [10,112,200,3]
np.save(file='data/calibdata.npy', arr=calib_datas)

loaded_data = np.load('data/calibdata.npy')
print(f'loaded_data.shape: {loaded_data.shape}') # [10,112,200,3]

"""
-cind INPUT_NAME NUMPY_FILE_PATH MEAN STD
int8_calib_datas = (loaded_data - MEAN) / STD # -1.0 - 1.0

e.g. How to specify calibration data in CLI or Script respectively.
1. CLI
  -cind "pc_dep" "data/calibdata.npy" "[[[[0.485,0.456,0.406]]]]" "[[[[0.229,0.224,0.225]]]]"
  -cind "feat" "data/calibdata2.npy" "[[[[0.123,...,0.321]]]]" "[[[[0.112,...,0.451]]]]"

2. Script
  custom_input_op_name_np_data_path=[
      ["pc_dep", "data/calibdata.npy", [[[[0.485,0.456,0.406]]]], [[[[0.229,0.224,0.225]]]]],
      ["feat", "data/calibdata2.npy", [[[[0.123,...,0.321]]]], [[[[0.112,...,0.451]]]],
  ]
"""

If you do not need to perform INT8 quantization with this tool alone, the following method is the easiest.

The -osd option will output a saved_model.pb in the saved_model folder with the full size required for quantization. That is, a default signature named serving_default is embedded in .pb. The -b option is used to convert the batch size by rewriting it as a static integer.

Note: INT8 TFLite generated by following this procedure as is will result in a model with significantly degraded accuracy. This tutorial only demonstrates the INT8 quantization procedure; if you wish to correct for accuracy, please refer to Parameter replacement to correct for transposition errors in the operation.

# Ref: https://github.com/onnx/models/tree/main/text/machine_comprehension/bert-squad
wget https://s3.ap-northeast-2.wasabisys.com/temp-models/onnx2tf_248/bertsquad-12.onnx

onnx2tf -i bertsquad-12.onnx -b 1 -osd -cotof

Use the saved_model_cli command to check the saved_model signature. INT8 quantization calibration using signatures allows correct control of the input order of data for calibration. Therefore, calibration with signatures is recommended for INT8 quantization of models with multiple inputs.

saved_model_cli show --dir saved_model/ --tag_set serve --signature_def serving_default

The given SavedModel SignatureDef contains the following input(s):
  inputs['input_ids_0'] tensor_info:
      dtype: DT_INT64
      shape: (1, 256)
      name: serving_default_input_ids_0:0
  inputs['input_mask_0'] tensor_info:
      dtype: DT_INT64
      shape: (1, 256)
      name: serving_default_input_mask_0:0
  inputs['segment_ids_0'] tensor_info:
      dtype: DT_INT64
      shape: (1, 256)
      name: serving_default_segment_ids_0:0
  inputs['unique_ids_raw_output___9_0'] tensor_info:
      dtype: DT_INT64
      shape: (1)
      name: serving_default_unique_ids_raw_output___9_0:0

Calibrate by specifying the input OP name displayed in inputs. The np.ones([xxx], dtype=np.int64) part must be replaced with the correct calibration test data. In practice, several pieces of data used for training are extracted and used.

import tensorflow as tf
import numpy as np

def representative_dataset():
    unique_ids = np.ones([10, 256], dtype=np.int64)
    segment_ids = np.ones([10, 256], dtype=np.int64)
    input_masks = np.ones([10, 256], dtype=np.int64)
    input_ids = np.ones([10], dtype=np.int64)

    for unique_id, segment_id, input_mask, input_id \
        in zip(unique_ids, segment_ids, input_masks, input_ids):

        yield {
            "unique_ids_raw_output___9_0": unique_id,
            "segment_ids_0": segment_id,
            "input_mask_0": input_mask,
            "input_ids_0": input_id,
        }

converter = tf.lite.TFLiteConverter.from_saved_model('saved_model')
converter.optimizations = [tf.lite.Optimize.DEFAULT]
converter.representative_dataset = representative_dataset
converter.target_spec.supported_ops = [tf.lite.OpsSet.TFLITE_BUILTINS_INT8]
converter.inference_input_type = tf.int8  # or tf.uint8
converter.inference_output_type = tf.int8  # or tf.uint8
tflite_quant_model = converter.convert()

with open('saved_model/int8_model.tflite', 'wb') as w:
    w.write(tflite_quant_model)

https://www.tensorflow.org/lite/performance/post_training_quantization

See: #248

PyTorch's NonMaxSuppression (torchvision.ops.nms) and ONNX's NonMaxSuppression are not fully compatible. TorchVision's NMS is very inefficient. Therefore, it is inevitable that converting ONNX using NMS in object detection models and other models will be very redundant and will be converted with a structure that is difficult for TensorFlow.js and TFLite models to take advantage of in devices. This is due to the indefinite number of tensors output by the NMS. In this chapter, I share how to easily tune the ONNX generated using TorchVision's redundant NMS to generate an optimized NMS.

1. There are multiple issues with TorchVision's NMS. First, the batch size specification is not supported; second, the max_output_boxes_per_class parameter cannot be specified. Please see the NMS sample ONNX part I generated. The max_output_boxes_per_class has been changed to 896 instead of -Infinity. The biggest problem with TorchVision NMS is that it generates ONNX with max_output_boxes_per_class set to -Infinity or 9223372036854775807 (Maximum value of INT64), resulting in a variable number of NMS outputs from zero to infinite. Thus, by rewriting -Infinity or 9223372036854775807 (Maximum value of INT64) to a constant value, it is possible to output an NMS that can be effortlessly inferred by TFJS or TFLite.

   Here you will find committed ONNX components optimized for various devices. https://github.com/PINTO0309/components_of_onnx/tree/main/components_of_onnx/ops

2. In the following example, the max_output_boxes_per_class of NMS in the post-processing generated by YOLOv7 is changed from -Infinity or 9223372036854775807 (Maximum value of INT64) to 20, as shown in the figure below. The name main01_max_output_boxes_per_class has been rewritten by me for clarity, but it originally appears as max_output_boxes_per_class.

   Simply execute the following command. The command rewrites the specified attribute value of the OP specified by ONNX.

   pip install sam4onnx
   
   sam4onnx \
   --op_name main01_nonmaxsuppression11 \
   --input_onnx_file_path yolov7.onnx \
   --output_onnx_file_path nms_yolov7_update.onnx \
   --input_constants main01_max_output_boxes_per_class int64 [20]

   A tutorial on one of my ONNX modification tools, sam4onnx, can be found here.

   https://github.com/PINTO0309/sam4onnx

   Many detailed tutorials are provided below, so if you are interested, please play with them.

   https://github.com/PINTO0309/PINTO_model_zoo/tree/main/307_YOLOv7/post_process_gen_tools

3. Finally, simply convert ONNX to TFLite or saved_model or TFJS using onnx2tf. onnx2tf performs an internal operation to automatically optimize the NMS output to a fixed shape if max_output_boxes_per_class is set to a value other than -Infinity and 9223372036854775807 (Maximum value of INT64). Specify --output_nms_with_dynamic_tensor or -onwdt if you do not want to optimize for a fixed shape.

   onnx2tf -i nms_yolov7_update.onnx -osd -cotof
   

   I would be happy if this is a reference for Android Java or TFJS implementations. There are tons more tricky model optimization techniques described in my blog posts, so you'll have to find them yourself. I don't dare to list the URL here because it is annoying to see so many issues being posted. And unfortunately, all articles are in Japanese.

TensorFlow's RNN has a speedup option called unroll. The network will be unrolled, else a symbolic loop will be used. Unrolling can speed-up a RNN, although it tends to be more memory-intensive. Unrolling is only suitable for short sequences. onnx2tf allows you to deploy RNNs into memory-intensive operations by specifying the --enable_rnn_unroll or -eru options. The --enable_rnn_unroll option is available for RNN, GRU, and LSTM.

• Keras https://keras.io/api/layers/recurrent_layers/lstm/
• TensorFlow https://www.tensorflow.org/api_docs/python/tf/keras/layers/LSTM

An example of BidirectionalLSTM conversion with the --enable_rnn_unroll option is shown below. Please ignore that the shapes of the input and output tensors do not match, since the samples are shown by picking up separate models.

• ONNX LSTM (Bidirectional)

  

• BidirectionalLSTM with --enable_rnn_unroll option unspecified

  Recurrent layer is implemented from scratch.

  

• BidirectionalLSTM with --enable_rnn_unroll option

  

The pattern of accuracy degradation of the converted model does not only occur when INT8 quantization is performed. A special edge case is when there is a problem with the implementation of a particular OP on the TFLite runtime side. Below, I will reproduce the problem by means of a very simple CNN model and further explain its workaround. Here is the issue that prompted me to add this explanation. [Conv-TasNet] Facing issue in converting Conv-TasNet model #447

Download a sample model for validation.

curl \
-L https://github.com/PINTO0309/onnx2tf/files/12367312/prelu_check.onnx.zip \
-o prelu_check.onnx.zip

unzip prelu_check.onnx.zip

The part of the downloaded model where the problem occurs is the PRelu part in the figure below.

• ONNX

  

Reproduce the problem. The following command converts an ONNX file to a TFLite file.

onnx2tf -i prelu_check.onnx -cotof

The conversion was successful and, as shown in the figure below, the inference test results from ONNX and the inference results for the Float32 model in TensorFlow (Keras) match perfectly. It is important to note that the comparison of inference results between ONNX and TensorFlow transformed models is comparing ONNX models with TensorFlow (Keras) models, not ONNX models with TFLite models.

• Conversion results

  

• tflite

  

Now, let's try inference with the TFLite runtime instead of the TensorFlow runtime.

• test.py

  import time
  import numpy as np
  np.random.seed(0)
  import tensorflow as tf
  
  # Load TFLite model
  interpreter = tf.lite.Interpreter(model_path="./saved_model/prelu_check_float32.tflite")
  interpreter.allocate_tensors()
  tensor_shape = (256, 20)
  input_data = {'waveform': np.random.randn(*tensor_shape).astype(np.float32)}
  
  # Load and preprocess
  input_details = interpreter.get_input_details()
  input_shape = input_details[0]['shape']
  print(input_shape)
  
  # Run inference
  interpreter.set_tensor(input_details[0]['index'], input_data["waveform"])
  separate_time = time.time()
  interpreter.invoke()
  print("Done! {:.3f} s".format(time.time() - separate_time))
  output_details = interpreter.get_output_details()
  output_data = interpreter.get_tensor(output_details[0]['index'])
  
  output_data = []
  for output_detail in output_details:
      output_data.append(interpreter.get_tensor(output_detail['index']))
  
  print(output_data)

Oddly enough, the output value of PReLU contains multiple nan. However, as can be seen by converting the ONNX model to the middle of the model using the -onimc option, nan does not occur until just before PReLU. Thus, it is clear that the PReLU OP in the TFLite runtime has a problem with divergent inference results.

• TFLite inference results

  

The following is a work-around to avoid this problem. Use the -rtpo option to replace PReLU with a similar primitive operation when transforming a model, and then perform the model transformation.

onnx2tf -i prelu_check.onnx -cotof -rtpo PReLU

As before, the inference results from ONNX and TensorFlow (Keras) match perfectly.

• Conversion results

  

However, -rtpo PReLU will generate a .tflite file with the PRelu OP replaced by a primitive OP combination.

• tflite

  

Again, run the test code to check the inference results. The figure below shows that no nan occurs when inference is performed by replacing the PReLU OP with only combinations of primitive operations. In other words, it is important to know that large arithmetic errors are not only due to the broken structure of the model, but can also be caused by internal implementations such as the TFLite runtime. I have implemented the -rtpo option to replace operators as a work-around to avoid such runtime problems.

• TFLite inference results

  

Even if the conversion is successful, InstanceNormalization tends to have very large errors. This is an ONNX specification.

• See.1: https://discuss.pytorch.org/t/understanding-instance-normalization-2d-with-running-mean-and-running-var/144139
• See.2: pytorch/pytorch#72057

I verified this with a very simple sample model. There are more than 8 million elements, and the calculation error reached 1e-2.

For some time now, TFLite runtime has supported inference by dynamic tensors. However, the existence of this important function is not widely recognized. In this chapter, I will show how I can convert an ONNX file that contains dynamic geometry in batch size directly into a TFLite file that contains dynamic geometry and then further infer it in variable batch conditions. The issue that inspired me to add this tutorial is here. [Dynamic batch / Dynamic shape] onnx model with dynamic input is converted to tflite with static input 1 #441, or Cannot use converted model with dynamic input shape #521

First, download the sample ONNX file.

wget https://s3.ap-northeast-2.wasabisys.com/temp-models/onnx2tf_441/osnet_x0_25_msmt17.onnx

This model calculates the similarity of features by cosine similarity. The batch size dimension of the input tensor is batch, allowing various numbers of images to be input simultaneously. This is often used, for example, to achieve tracking by calculating the similarity of people or objects reflected between successive video frames. However, the total number of objects to be tracked changes rapidly with each video frame because the number of people and objects in the image constantly increases and decreases. Therefore, there is a very significant use case for generating models with variable settings for the number of input images (batch size) of the model.

Convert the downloaded OSNet to tflite and saved_model as a variable batch. If you do not specify the -b or -ois options, onnx2tf does not change the batch size as N. The only important point is to convert the model with the -osd and -coion options. Note that if you use the -coion option, you must install flatbuffers-compiler with apt-get install, run the commands for building the environment described first in this README, or use a Docker container.

onnx2tf -i osnet_x0_25_msmt17.onnx -osd -coion

• .tflite

  When viewing tflite in Netron, the batch size appears to be fixed at 1.

• saved_model

  However, checking the structure of saved_model, the batch size is correctly set to -1.

  saved_model_cli show --dir saved_model/ --all
  
  MetaGraphDef with tag-set: 'serve' contains the following SignatureDefs:
  
  signature_def['__saved_model_init_op']:
    The given SavedModel SignatureDef contains the following input(s):
    The given SavedModel SignatureDef contains the following output(s):
      outputs['__saved_model_init_op'] tensor_info:
          dtype: DT_INVALID
          shape: unknown_rank
          name: NoOp
    Method name is:
  
  signature_def['serving_default']:
    The given SavedModel SignatureDef contains the following input(s):
      inputs['images'] tensor_info:
          dtype: DT_FLOAT
          shape: (-1, 256, 128, 3)
          name: serving_default_images:0
    The given SavedModel SignatureDef contains the following output(s):
      outputs['output'] tensor_info:
          dtype: DT_FLOAT
          shape: (-1, 512)
          name: PartitionedCall:0
    Method name is: tensorflow/serving/predict

To prove that the tflite structure has been converted correctly, I will convert the tflite to JSON and look at the structure.

docker run --rm -it \
-v `pwd`:/home/user/workdir \
ghcr.io/pinto0309/tflite2json2tflite:latest

./flatc -t \
--strict-json \
--defaults-json \
-o workdir \
./schema.fbs -- workdir/saved_model/osnet_x0_25_msmt17_float32.tflite

ls -l workdir

-rw-rw-r-- 1 user user   921564 Aug  4 10:24 osnet_x0_25_msmt17.onnx
-rw-r--r-- 1 user user 10369524 Aug  4 10:30 osnet_x0_25_msmt17_float32.json
drwxrwxr-x 4 user user     4096 Aug  4 10:26 saved_model

• osnet_x0_25_msmt17_float32.json

  "shape_signature" is correctly set to -1. However, "shape" is set to 1. This could be a problem with TFLiteConverter, or it could be a problem with Netron's graphical display capabilities.

In other words, although onnx2tf converts TFLiteConverer as specified, with the batch size of -1 without any model processing, only Netron's display is broken. This is a problem I have known for quite some time. However, the inference itself does not cause the problem.

If you want to infer in variable batches, you need to infer using signature. In such cases, the -coion option must be specified when converting the model. Note that I have identified a problem with quantization with the -coion option, which can corrupt tflite files. #429

https://github.com/PINTO0309/onnx2tf#4-match-tflite-inputoutput-names-and-inputoutput-order-to-onnx

You can use signature_runner to handle dynamic input tensors by performing inference using signature. Below I show that both batch_size=5 and batch_size=3 tensors can be inferred with the same model.

• test.py - Batch size: 5

  import numpy as np
  import tensorflow as tf
  from pprint import pprint
  
  interpreter = tf.lite.Interpreter(model_path="saved_model/osnet_x0_25_msmt17_float32.tflite")
  tf_lite_model = interpreter.get_signature_runner()
  inputs = {
      'images': np.ones([5,256,128,3], dtype=np.float32),
  }
  tf_lite_output = tf_lite_model(**inputs)
  print(f"[TFLite] Model Predictions shape: {tf_lite_output['output'].shape}")
  print(f"[TFLite] Model Predictions:")
  pprint(tf_lite_output)

• Results

  [TFLite] Model Predictions shape: (5, 512)
  [TFLite] Model Predictions:
  {'output': array([[0.0000000e 00, 2.4730086e-04, 0.0000000e 00, ..., 1.0528549e 00,
          3.7874988e-01, 0.0000000e 00],
         [0.0000000e 00, 2.4730086e-04, 0.0000000e 00, ..., 1.0528549e 00,
          3.7874988e-01, 0.0000000e 00],
         [0.0000000e 00, 2.4730086e-04, 0.0000000e 00, ..., 1.0528549e 00,
          3.7874988e-01, 0.0000000e 00],
         [0.0000000e 00, 2.4730086e-04, 0.0000000e 00, ..., 1.0528549e 00,
          3.7874988e-01, 0.0000000e 00],
         [0.0000000e 00, 2.4730084e-04, 0.0000000e 00, ..., 1.0528525e 00,
          3.7874976e-01, 0.0000000e 00]], dtype=float32)}
  

• test.py - Batch size: 3

  import numpy as np
  import tensorflow as tf
  from pprint import pprint
  
  interpreter = tf.lite.Interpreter(model_path="saved_model/osnet_x0_25_msmt17_float32.tflite")
  tf_lite_model = interpreter.get_signature_runner()
  inputs = {
      'images': np.ones([3,256,128,3], dtype=np.float32),
  }
  tf_lite_output = tf_lite_model(**inputs)
  print(f"[TFLite] Model Predictions shape: {tf_lite_output['output'].shape}")
  print(f"[TFLite] Model Predictions:")
  pprint(tf_lite_output)

• Results

  [TFLite] Model Predictions shape: (3, 512)
  [TFLite] Model Predictions:
  {'output': array([[0.0000000e 00, 2.4730084e-04, 0.0000000e 00, ..., 1.0528525e 00,
          3.7874976e-01, 0.0000000e 00],
         [0.0000000e 00, 2.4730084e-04, 0.0000000e 00, ..., 1.0528525e 00,
          3.7874976e-01, 0.0000000e 00],
         [0.0000000e 00, 2.4730084e-04, 0.0000000e 00, ..., 1.0528525e 00,
          3.7874976e-01, 0.0000000e 00]], dtype=float32)}
  

Einsum and OneHot are not optimized to the maximum by the standard behavior of onnx-optimizer. Therefore, pre-optimizing the Einsum OP and OneHot OP using my original method can significantly improve the success rate of model conversion, and the input ONNX model itself can be significantly optimized compared to when onnxsim alone is optimized. See: #569

• I have made a few unique customizations to the cited model structure.

  https://github.com/PINTO0309/LightGlue-ONNX

• spo4onnx

  https://github.com/PINTO0309/spo4onnx

For example

python export.py \
--img_size 512 512 \
--lightglue_path weights/sjy_fused_static.onnx \
--end2end

pip install -U spo4onnx onnx2tf

cd weights
spo4onnx -if sjy_fused_static.onnx -of sjy_fused_static_spo.onnx

onnx2tf -i sjy_fused_static_spo.onnx

Sometimes you want to always output constants that are not connected to the model body. See: #627. For example, in the case of ONNX as shown in the figure below. You may want to keep scaling parameters and other parameters as fixed values inside the model and always include the same value in the output.

In such cases, the process of optimizing the ONNX file in onnxsim must be bypassed and not executed. You can bypass the execution of onnxsim by specifying -nuo or --not_use_onnxsim as a conversion option. Running onnxsim will remove constants from the model definition that are not connected to the body of the model in the process of optimizing the model structure.

wget https://github.com/PINTO0309/onnx2tf/files/15292126/toy_with_constant.onnx.zip
unzip toy_with_constant.onnx.zip
onnx2tf -i toy_with_constant.onnx -nuo -cotof

The relationship between the ONNX before conversion and the TFLite file after conversion is shown in the figure below.

ONNX	TFLite

Use the generated TFLite file to inference and ensure that it always contains fixed value output.

import tensorflow as tf
import numpy as np
from pprint import pprint

interpreter = tf.lite.Interpreter(model_path="saved_model/toy_with_constant_float32.tflite")
interpreter.allocate_tensors()

input_details = interpreter.get_input_details()
output_details = interpreter.get_output_details()

interpreter.set_tensor(
    tensor_index=input_details[0]['index'],
    value=np.ones(tuple(input_details[0]['shape']), dtype=np.float32)
)
interpreter.invoke()

variable_output = interpreter.get_tensor(output_details[0]['index'])
constant_output = interpreter.get_tensor(output_details[1]['index'])

print("=================")
print("Variable Output:")
pprint(variable_output)
print("=================")
print("Constant Output:")
pprint(constant_output)

=================
Variable Output:
array([[-0.02787317, -0.05505124,  0.05421712,  0.03526559, -0.14131774,
         0.0019211 ,  0.08399964,  0.00433664, -0.00984338, -0.03370604]],
      dtype=float32)
=================
Constant Output:
array([1., 2., 3., 4., 5.], dtype=float32)

This refers to a model with undefined dimensions, either all dimensions or multiple dimensions including batch size, as shown in the figure below.

• Sample model

  https://github.com/PINTO0309/onnx2tf/releases/download/1.24.0/bge-m3.onnx

• Structure

  

If such a model is converted without any options, TensorFlow/Keras will abort. This is an internal TensorFlow/Keras implementation issue rather than an onnx2tf issue. TensorFlow/Keras does not allow more than two undefined dimensions in the shape attribute of Reshape due to the specification, so an error occurs during the internal transformation operation of the Reshape OP as shown below. This has been an inherent problem in TensorFlow/Keras since long ago and has not been resolved to this day. See: RuntimeError: tensorflow/lite/kernels/range.cc:39 (start > limit && delta < 0) || (start < limit && delta > 0) was not true.Node number 3 (RANGE) failed to invoke. Node number 393 (WHILE) failed to invoke. current error :RuntimeError: tensorflow/lite/kernels/reshape.cc:55 stretch_dim != -1 (0 != -1)Node number 83 (RESHAPE) failed to prepare. #40504

• OP where the problem occurs

  

• Error message

  error: 'tf.Reshape' op requires 'shape' to have at most one dynamic dimension, but got multiple dynamic dimensions at indices 0 and 3
  

Thus, for models such as this, where all dimensions, including batch size, are dynamic shapes, it is often possible to convert by fixing the batch size to 1 with the -b 1 or --batch_size 1 option.

onnx2tf -i model.onnx -b 1 -osd

• Results

  

  When the converted tflite is displayed in Netron, all the dimensions of the dynamic shape are displayed as 1, but this is a display problem in Netron, and the shape is actually converted to -1 or None.

  

Click here to see how to perform inference using the dynamic shape tensor.

https://github.com/PINTO0309/onnx2tf/tree/main?tab=readme-ov-file#14-inference-with-dynamic-tensors-in-tflite

By specifying ONNX input or output names, only the middle part of the model can be converted. This is useful when you want to see what output is obtained in what part of the model after conversion, or when debugging the model conversion operation itself.

For example, take a model with multiple inputs and multiple outputs as shown in the figure below to try a partial transformation.

• To convert by specifying only the input name to start the conversion

  wget https://github.com/PINTO0309/onnx2tf/releases/download/1.25.0/cf_fus.onnx
  onnx2tf -i cf_fus.onnx -inimc 448 -coion
  

  

• To convert by specifying only the output name to end the conversion

  wget https://github.com/PINTO0309/onnx2tf/releases/download/1.25.0/cf_fus.onnx
  onnx2tf -i cf_fus.onnx -onimc dep_sec -coion
  

  

• To perform a conversion by specifying the input name to start the conversion and the output name to end the conversion

  wget https://github.com/PINTO0309/onnx2tf/releases/download/1.25.0/cf_fus.onnx
  onnx2tf -i cf_fus.onnx -inimc 448 -onimc velocity -coion
  

  

When converting to TensorFlow.js, process as follows.

pip install -U --no-deps \
tensorflowjs \
tensorflow_decision_forests \
ydf \
tensorflow_hub

onnx2tf -i mobilenetv2-12.onnx -ois input:1,3,224,224 -osd -dgc

tensorflowjs_converter \
--input_format tf_saved_model \
--output_format tfjs_graph_model \
saved_model \
tfjs_model

See: https://github.com/tensorflow/tfjs/tree/master/tfjs-converter

When converting to CoreML, process as follows. The -k option is for conversion while maintaining the input channel order in ONNX's NCHW format.

pip install coremltools

onnx2tf -i mobilenetv2-12.onnx -k input -ois input:1,3,224,224 -osd

import coremltools as ct

FOLDER_PATH = 'saved_model'

model = ct.convert(
    model=FOLDER_PATH,
    source='tensorflow',
)
model.save(f'{FOLDER_PATH}/model.mlmodel')

See: https://github.com/apple/coremltools

Starting from v1.3.0, almost all OPs except for some special OPs support pre- and post-transposition by pre_process_transpose and post_process_transpose.

1. "A conversion error occurs."
2. "Output results are wrong."

Do not submit an issue that only contains an amount of information that cannot be reproduced.

• convert option

  --param_replacement_file param_replacement.json
  
  or
  
  -prf param_replacement.json
  

• param_replacement.json

  See a sample of replacement JSON

  {
    "format_version": 1,
    "operations": [
      {
        "op_name": "StatefulPartitionedCall/Tile_4",
        "param_target": "inputs", # attributes or inputs
        "param_name": "const_fold_opt__677",
        "values": [1,1,17] # Disable parameter transposition or overwrite parameters
      },
      {
        "op_name": "StatefulPartitionedCall/Cast_3",
        "param_target": "attributes", # attributes or inputs
        "param_name": "to",
        "values": 1 # Disable parameter transposition or overwrite "to" parameters
      },
      {
        "op_name": "Resize__697",
        "param_target": "inputs",
        "param_name": "Concat__696:0",
        "values": [26,26] # Replacement of unk__x (Resize OP, sizes height/width parameter)
      },
      {
        "op_name": "Transpose__927",
        "param_target": "attributes",
        "param_name": "perm",
        "values": [0,1,2,3] # Disable parameter transposition or overwrite "perm" parameters
      },
      {
        "op_name": "StatefulPartitionedCall/functional_1/max_unpooling2d_2/Reshape_1",
        "param_target": "inputs",
        "param_name": "const_fold_opt__911",
        "values": [4,131072] # Overwrite "shape" parameters
      },
      {
        "op_name": "Reshape_25",
        "param_target": "outputs",
        "param_name": "onnx::InstanceNormalization_270",
        "post_process_transpose_perm": [0,2,1] # Extrapolate 3D Transpose after Reshape
      },
      {
        "op_name": "Reshape_30",
        "param_target": "outputs",
        "param_name": "onnx::Mul_275",
        "post_process_transpose_perm": [0,2,3,1] # Extrapolate 4D Transpose after Reshape
      },
      {
        "op_name": "flatten_1127",
        "param_target": "inputs",
        "param_name": "dropout0",
        "pre_process_transpose_perm": [0,3,1,2]
      },
      {
        "op_name": "/Slice",
        "param_target": "op",
        "begin": [0,0,1,0],
        "end": [0,0,0,0],
        "end_mask": 15
      },
      {
        "op_name": "/Slice_1",
        "param_target": "op",
        "begin": [0,0,0,0],
        "end": [0,0,39,0],
        "end_mask": 11
      },
      {
        "op_name": "/backbone/backbone.1/Unsqueeze_1",
        "param_target": "op",
        "new_shape": [1,15,15,1]
      }
    ]
  }

• Replacement Supported OPs

  See list of replacement specifications

  No.	OP type	Remarks
  1	Add	1. "param_target": "inputs" pre_process_transpose_perm: Transpose is applied to the tensor before the Add operation with the perm specified as pre-processing. 2. "param_target": "outputs" post_process_transpose_perm: Transpose is applied to the tensor after the Add operation with the perm specified as post-processing.
  2	Cast	Type Values Type Values float16 10 int8 3 float32 1 int16 5 float64 11 int32 6 bool 9 int64 7 uint8 2 uint16 4 uint32 12 uint64 13
  3	Concat	1. "param_target": "attributes" axis: Value of axis 2. "param_target": "outputs" post_process_transpose_perm: Transpose is applied to the tensor after the Concat operation with the perm specified as post-processing.
  4	ConvTranspose	ConvTranspose implements special replacements separately ignore all automatic conversions and generate tf.nn.conv1d_transpose or tf.nn.conv2d_transpose or tf.nn.conv3d_transpose directly by specifying all parameters. https://www.tensorflow.org/api_docs/python/tf/nn/conv1d_transpose https://www.tensorflow.org/api_docs/python/tf/nn/conv2d_transpose https://www.tensorflow.org/api_docs/python/tf/nn/conv3d_transpose 1. "param_target": "op" output_shape: Value of output_shape strides: Value of strides padding: Value of padding dilations: Value of dilations
  5	Div	1. "param_target": "inputs" values: Value of input pre_process_transpose_perm: Transpose is applied to the tensor before the Div operation with the perm specified as pre-processing. 2. "param_target": "outputs" post_process_transpose_perm: Transpose is applied to the tensor after the Div operation with the perm specified as post-processing.
  6	Expand	1. "param_target": "inputs" values: Value of shape pre_process_transpose_perm: Transpose is applied to the tensor before the Expand operation with the perm specified as pre-processing. 2. "param_target": "outputs" post_process_transpose_perm: Transpose is applied to the tensor after the Expand operation with the perm specified as post-processing.
  7	Flatten	1. "param_target": "attributes" axis: Value of axis 2. "param_target": "inputs" pre_process_transpose_perm: Transpose is applied to the tensor before the Flatten operation with the perm specified as pre-processing. 3. "param_target": "outputs" post_process_transpose_perm: Transpose is applied to the tensor after the Flatten operation with the perm specified as post-processing.
  8	Gemm
  9	Gather	1. "param_target": "attributes" axis: Value of axis 2. "param_target": "inputs" values: Value of indices pre_process_transpose_perm: Transpose is applied to the tensor before the Gather operation with the perm specified as pre-processing. 3. "param_target": "outputs" post_process_transpose_perm: Transpose is applied to the tensor after the Gather operation with the perm specified as post-processing.
  10	MatMul	1. "param_target": "inputs" pre_process_transpose_perm: Transpose is applied to the tensor before the MatMul operation with the perm specified as pre-processing. 2. "param_target": "outputs" post_process_transpose_perm: Transpose is applied to the tensor after the MatMul operation with the perm specified as post-processing.
  11	Mul	1. "param_target": "inputs" values: Value of input pre_process_transpose_perm: Transpose is applied to the tensor before the Mul operation with the perm specified as pre-processing. 2. "param_target": "outputs" post_process_transpose_perm: Transpose is applied to the tensor after the Mul operation with the perm specified as post-processing.
  12	NonMaxSuppression
  13	ReduceL1 ReduceL2 ReduceLogSum ReduceLogSumExp ReduceMax ReduceMean ReduceMin ReduceProd ReduceSum ReduceSumSquare	1. "param_target": "attributes" axes: Value of axes keepdims: Value of keepdims 2. "param_target": "inputs" pre_process_transpose_perm: Transpose is applied to the tensor before the ReduceXX operation with the perm specified as pre-processing. 3. "param_target": "outputs" post_process_transpose_perm: Transpose is applied to the tensor after the ReduceXX operation with the perm specified as post-processing.
  14	Unsqueeze	1. "param_target": "inputs" pre_process_transpose_perm: Transpose is applied to the tensor before the Unsqueeze operation with the perm specified as pre-processing. 2. "param_target": "outputs" post_process_transpose_perm: Transpose is applied to the tensor after the Unsqueeze operation with the perm specified as post-processing. 3. "param_target": "op" new_shape: Specifies directly the shape after Unsqueeze processing. {   "op_name": "/backbone/backbone.1/Unsqueeze_1",   "param_target": "op",   "new_shape": [1,15,15,1] }
  15	Reshape	1. "param_target": "inputs" values: Value of shape pre_process_transpose_perm: Transpose is applied to the tensor before the Reshape operation with the perm specified as pre-processing. 2. "param_target": "outputs" post_process_transpose_perm: Transpose is applied to the tensor after the Reshape operation with the perm specified as post-processing.
  16	Resize	1. "param_target": "attributes" coordinate_transformation_mode: Value of coordinate_transformation_mode extrapolation_value: Value of extrapolation_value mode: Value of mode 2. "param_target": "inputs" values: Value of roi or scales or sizes. scales=[scale_h,scale_w],sizes=[h,w] pre_process_transpose_perm: Transpose is applied to the tensor before the Resize operation with the perm specified as pre-processing. 3. "param_target": "outputs" post_process_transpose_perm: Transpose is applied to the tensor after the Resize operation with the perm specified as post-processing.
  17	Slice	Slice implements special replacements separately ignore all automatic conversions and generate tf.strided_slice directly by specifying all parameters of tf.strided_slice directly. https://www.tensorflow.org/api_docs/python/tf/strided_slice See json_samples/replace_slice.json for a sample description. 1. "param_target": "op" begin: Value of begin end: Value of end strides: Value of strides begin_mask: Value of begin_mask end_mask: Value of end_mask ellipsis_mask: Value of ellipsis_mask new_axis_mask: Value of new_axis_mask shrink_axis_mask: Value of shrink_axis_mask {   "op_name": "/Slice",   "param_target": "op",   "begin": [0,0,1,0],   "end": [0,0,0,0],   "end_mask": 15 }
  18	Softmax	1. "param_target": "attributes" axis: Value of axis. The transpositions corresponding to the specified axis are extrapolated before and after Softmax. 2. "param_target": "inputs" values: Value of tensor
  19	Split	1. "param_target": "inputs" values: Value of split 2. "param_target": "attributes" axis: Value of axis. num_outputs: Value of num_outputs.
  20	Sub	1. "param_target": "inputs" values: Value of input pre_process_transpose_perm: Transpose is applied to the tensor before the Sub operation with the perm specified as pre-processing. 2. "param_target": "outputs" post_process_transpose_perm: Transpose is applied to the tensor after the Sub operation with the perm specified as post-processing.
  21	Tile	1. "param_target": "inputs" values: Value of input pre_process_transpose_perm: Transpose is applied to the tensor before the Tile operation with the perm specified as pre-processing. 2. "param_target": "outputs" post_process_transpose_perm: Transpose is applied to the tensor after the Tile operation with the perm specified as post-processing.
  22	Transpose	1. "param_target": "attributes" perm: Value of perm 2. "param_target": "inputs" values: Value of tensor

• onnx-tensorflow is a very useful tool, but the performance of the generated TensorFlow models is significantly degraded due to the extrapolation of a large number of Transpose OPs before and after each OP during the format conversion from NCHW to NHWC. Therefore, I will make this tool myself as a derivative tool of onnx-tensorflow without extrapolating Transpose.

• Most of the internal processing of the tool is full-scratch, but some of the more complex OPs have been adapted from onnx-tensorflow. I am very grateful to the engineers at International Business Machines Corporation / LeapMind / Microsoft / IBM for developing onnx-tensorflow.

• I have incorporated all my knowledge of model optimization to other models such as TFLite, EdgeTPU, TensorFlow.js and Myriad based on my years of experience implementing openvino2tensorflow and tflite2tensorflow. It probably has the best model optimization performance and conversion efficiency of any tool I have created in the past, and the lowest rate of conversion errors.

• Supported layers list. Supported layers

• If you are having trouble with conversion errors, searching for resolved or open issues will almost always solve your problems. Issues are knowledge for engineers around the world.

• Contributors to this repository should first read Contribution Guide.

  Kazam_screencast_00065_.mp4

• All OPs are decomposed into primitive operations as much as possible. This is beneficial for lateral deployment of models to frameworks other than TFLite. Therefore, OPs belonging to tf_keras.layers are almost never used, and the tool consists only of tf.xxx. (except for a very few OPs)

• As I do not want to add more dependent packages, I do not use tensorflow_addons (tfa), but replace it with the standard OP of tensorflow.

• Not only does it handle conversions of 4-dimensional inputs, such as NCHW to NHWC, but also the number of input dimensions in 3, 5, or even more dimensions. For example, NCDHW to NDHWC, etc. However, since 1-D, 2-D, 3-D and 6-D input may produce patterns that are mechanically difficult to convert, it should be possible to give parameters to externally modify the tool's behavior. See Parameter replacement

• If there are undefined dimensions in the input OP, the model structure is not fully optimized and conversion errors are very likely to occur.

• Immediately following a Reshape OP with dimensional compression and dimensional decompression, there is a 95% probability that the model transformation operation will be disrupted and errors will occur. For example, patterns such as [1,200,200,5] -> [1,200,-1] or [10,20,30,40,50] -> [10,2,10,30,10,4,50] or Flatten. See #8 Not able to reshape input in replace.json, or #15 Conv layer shape wrong, or #18 Question about channel_transpose in common_functions.py, or #105 [MobileFormer]Converted model outputs values mismatch with original ones., or #133 When Onnx Matmul inputs have different dimension.

• TensorFlow's Convolution does not have an equivalent operation to ONNX's Padding operation. Therefore, a Pad OP is inserted immediately before a Convolution with Padding of size greater than 1.

• Support conversion to TensorFlow saved model and TFLite (Float32/Float16/INT8).

• Files exceeding the Protocol Buffers file size limit of 2GB are not supported. Therefore, the external format is not supported at the initial stage of tool creation.

• If there are ONNX OPs that are not supported by TensorFlow, use simple-onnx-processing-tools to replace them with harmless OPs in advance and then use this tool to convert them. In other words, you can convert any model with your efforts.

• ONNX splitting, merging, generating OPs, rewriting OP attributes, BGR<->RGB conversion, converting to JSON and editing in the IDE, batch size changes for undefined dimensions, and various other processing can be done with the simple-onnx-processing-tools. Therefore, it is recommended that models with very complex structures be converted to TFLite after modifying the structure beforehand.

• BatchNormalization supports only inference mode.

• LayerNormalization supports only inference mode.

• Only for opset=11 or higher

• If you do not like the generated TFLite OP name, edit it using tflite2json2tflite.

• The generated Keras models cannot be used for retraining. If you want to train, you must build your own model.

• When converting to TensorFlow.js, CoreML, etc., please generate saved_model with the --output_signaturedefs option and use the generated saved_model to convert with various converters. tensorflowjs_converter, coremltools, edgetpu_compilier, etc... If this option is not enabled, saved_model records only the minimum necessary information and its size is minimized. When this option is enabled, saved_model records the maximum amount of information, and instead of being maximized in size, the output is in a format that supports conversion to other frameworks. It can also be used for serving.

• There are many OPs on ONNX that do not support TFLite/EdgeTPU/TFJS/CoreML/TensorRT. Therefore, if you need to generate an EdgeTPU model, please specify --replace_to_pseudo_operators to convert your model. onnx2tf will attempt to replace the OP with an TFLite/EdgeTPU/TFJS/CoreML/TensorRT-compatible OP whenever possible.

• The main factors that cause accuracy degradation after model conversion are as follows

1. differences in Padding specifications
2. difference in Python division specification in the process of model transformation (error due to even rounding)
3. Divide epsilon without consideration
4. deprecated TrueDivision
5. support difference of powers
6. differences in interpolation operation specifications during resizing
7. Difference in arithmetic precision supported by each operation
8. Calculation error due to scaling up or down by specifying a scale when resizing images

The above differences often cannot be dealt with by simply converting the model in a straightforward manner. Therefore, you need to replace the model yourself in advance with an operation that is less prone to errors.

• Support for INT8 Quantization, Full INT8 Quantization, INT8 Quantization with INT16 activation, Full INT8 Quantization with INT16 activation and Dynamic Range Quantization.
• Support for Per-Channel Quantization and Per-Tensor Quantization.
• Support for GroupConvolution.
• TFLite does not support TrueDiv(INT), so TrueDiv is avoided if possible.
• Implement the Resize process for the 5D tensor.
• Add process to replace Asin with pseudo-Asin.
• Add process to replace Acos with pseudo-Acos.
• Add process to replace Atan with pseudo-Atan.
• Add process to replace Abs with pseudo-Abs.
• Add process to replace GatherND with pseudo-GatherND.
• Add process to replace HardSwish with pseudo-HardSwish.
• Add process to replace GridSample with pseudo-GridSample.
• Add process to replace PRelu with pseudo-PRelu.
• Add process to replace LeakyRelu with pseudo-LeakyRelu.
• Add process to replace Power with pseudo-Power.
• Add process to replace Neg with pseudo-Neg.
• Add process to replace ArgMax with pseudo-ArgMax.
• Add process to replace Erf with pseudo-Erf.
• Add process to replace GeLU with pseudo-GeLU.
• Added option to fix dynamic batch size N to a specified number.
• Added option to overwrite dynamic shape input OPs with static shape. --overwrite_input_shape
• Output in Keras H5 format.
• Automatically run onnx-simplifier (onnxsim) backend and optimize onnx files before model transformation.
• Added the ability to automatically generate each OP name and assign OP names to ONNX files in the old format.
• Supports model splitting. Interrupts model transformation at the specified output name and outputs the model partitioned into subgraphs.