What is the specific calculation formula for latency (ms/token)? Is it only the decoding time that needs to be calculated, or does the pre-filling time also need to be taken into account?

[canjie-liu]

[Gumpest]

Hi, @canjie-liu

We have already integrated the code for efficiency analysis in the repository.

When you run our latest version, the model will print the real-time efficiency metrics (including Equal Tokens, Prefill Time (ms), and TFLOPs) directly in the log during inference. This will allow you to monitor and analyze the computational cost and savings dynamically.