Performance issues about tf.function #58
Description
Hello! Our static bug checker has found a performance issue in ONNX/lambda-onnx/onnxruntime/transformers/benchmark.py: run_with_tf_optimizations (1),(2),(3) is repeatedly called in a for loop, but there is a tf.function decorated function run_in_graph_mode defined and called in run_with_tf_optimizations.
In that case, when run_with_tf_optimizations is called in a loop, the function run_in_graph_mode will create a new graph every time, and that can trigger tf.function retracing warning.
Similar problems in ONNX-ARM/lambda-onnx-arm-3.8/onnxruntime/transformers/benchmark.py.
Here is the tensorflow document to support it.
Briefly, for better efficiency, it's better to use:
@tf.function
def inner():
pass
def outer():
inner() than:
def outer():
@tf.function
def inner():
pass
inner()Looking forward to your reply.