BEVFormer量化笔记4

本篇接续笔记 3:性能分析(第 7 节)与 layer info 分析(第 8 节)。

7. 性能分析

结论:INT8 相比 FP16 没有性能收益,精度却差得多(见 7.2);FP16 吞吐约为 FP32 的 1.9 倍,是后续改进的方向。

7.1. 性能

INT8性能并未提升,且模型本身受Orin资源占用影响,耗时波动大

performanceFP32高负载FP16低负载FP16高负载INT8低负载INT8高负载
Throughput2.6826 qps5.0788 qps2.7949 qps5.0647 qps2.6911 qps
Latency (max/mean/min)382.755/372.773/359.955 ms203.397/196.897/190.550 ms384.442/357.798/316.573 ms202.538/197.446/195.458 ms419.296/371.597/324.387 ms
Latency median373.278 ms198.540 ms364.538 ms197.350 ms375.708 ms
Latency percentile(90%)379.852 ms201.924 ms374.321 ms199.032 ms399.520 ms
Latency percentile(95%)380.251 ms202.654 ms375.742 ms199.412 ms401.939 ms
Latency percentile(99%)381.929 ms203.358 ms377.053 ms201.150 ms416.175 ms
H2D Latency (max/mean/min)4.605/3.124/2.183 ms3.961/3.167/2.868 ms3.857/2.707/2.222 ms4.959/3.125/2.753 ms5.729/3.133/2.331 ms
H2D Latency median3.029 ms3.094 ms2.626 ms3.026 ms2.964 ms
H2D Latency percentile(90%)3.743 ms3.440 ms3.195 ms3.507 ms4.104 ms
H2D Latency percentile(95%)4.105 ms3.587 ms3.363 ms3.638 ms4.646 ms
H2D Latency percentile(99%)4.197 ms3.837 ms3.706 ms4.707 ms5.531 ms
GPU Compute Time (max/mean/min)378.690/369.103/356.528 ms199.926/193.127/187.139 ms380.242/353.396/313.835 ms197.329/193.987/192.298 ms410.706/367.622/321.857 ms
GPU Compute Time median369.869 ms194.784 ms359.157 ms193.899 ms371.218 ms
GPU Compute Time percentile(90%)376.354 ms198.015 ms371.048 ms195.617 ms396.222 ms
GPU Compute Time percentile(95%)377.252 ms198.908 ms372.035 ms196.021 ms398.074 ms
GPU Compute Time percentile(99%)378.442 ms199.774 ms372.940 ms196.821 ms408.112 ms
D2H Latency (max/mean/min)4.944/0.447/0.104 ms1.219/0.505/0.136 ms9.832/1.603/0.107 ms1.159/0.232/0.121 ms7.631/0.743/0.103 ms
D2H Latency median0.434 ms0.495 ms0.628 ms0.166 ms0.420 ms
D2H Latency percentile(90%)0.863 ms0.872 ms5.198 ms0.496 ms1.188 ms
D2H Latency percentile(95%)1.119 ms0.903 ms7.693 ms0.555 ms3.149 ms
D2H Latency percentile(99%)1.440 ms1.179 ms9.814 ms0.782 ms5.623 ms
Total Host Walltime37.277 s19.690 s35.780 s19.745 s37.160 s
Total GPU Compute Time36.910 s19.313 s35.340 s19.399 s36.762 s

从表里能读出三件事:

  1. 低负载下 FP16 与 INT8 几乎一样快(吞吐 5.08 vs 5.06 qps,约为 FP32 的 1.9 倍),INT8 没有带来额外收益——与第 8 节「层数膨胀淹没计算加速」的分析一致。
  2. 高负载下三种精度都掉到 2.7–2.8 qps,且延迟 max/min 差距拉大,说明瓶颈在 Orin 资源争用,不在计算精度。
  3. H2D + D2H 中位数合计不到 4 ms,相对 200/370 ms 量级的 GPU Compute Time 可以忽略,耗时基本全在 GPU 计算上。

7.2. 精度

FP16精度需要改进,INT8精度过低。

precisionFP32FP16INT8
bbox_preds_cosine0.9998870.9995900.998596
cls_scores_cosine0.9260670.8317270.357765
bbox_preds_diff15.084946/0.134497/0.00000017.616318/0.241345/0.00000024.715466/0.540364/0.000000
cls_scores_diff0.945730/0.000194/0.0000000.938262/0.000326/0.0000000.978774/0.000642/0.000000

8. 分析layer info

错了错了,拿到模型之后应该先分析layer info!!!导出的模型layer可能有精度回退……

结论:FP16和INT8出现了严重了层数膨胀,膨胀源是非计算层的爆炸式增长,完全淹没了精度降低带来的计算加速。

指标FP32FP16INT8
总层数99488 (×5)1753 (×18)
计算层占比82.8%23.8%28.1%
Reformat 层523144
Slice + NoOp + Fill7323775
FP32 回退090

8.1. FP16分析

8.1.1. 结构膨胀

FP16新增了389层!

类型增量来源
Slice+154动态shape padding mask / stride 展开
NoOp+99Reshape / Unsqueeze 显式化
Fill+63动态 shape 下的 Linspace 索引生成
其他+73ElementWise、Shuffle、Myelin 等

8.1.2. FP32回退

Slice_2584 ────[FP16→FP32]───┐
FP16 输入1 ────[FP16→FP32]────┤
FP16 输入2 ────[FP16→FP32]────┤──→ ForeignNode[Add_2568...Unsqueeze_2943]
FP16 输入3 ────[FP16→FP32]────┤      ↑ Myelin 执行,FP32
FP16 输入4 ────[FP16→FP32]────┘
Reshape_25 ──────────────────┘ (NoOp, FP32)

一个 Myelin ForeignNode 导致 9 层回退。Myelin 是 TRT 的自动融合引擎,它把 Add + Unsqueeze 融合成了一个 ForeignNode,但这个融合后的算子不支持 FP16,被迫回退到 FP32,并且所有输入需要 FP16→FP32 的 reformat。

8.2. INT8分析

8.2.1. 结构膨胀

INT8膨胀到了1753层,FP32只有99层,膨胀了17.7倍

层类型数量占比说明
NoOp55631.7%Reshape/Unsqueeze 显式化
PointWiseV224714.1%逐元素操作(Q/DQ 相关)
Constant19311.0%量化权重常量
Slice1568.9%Padding mask
Reformat1448.2%FP32↔INT8 转换
CaskConvolution1418.0%实际卷积计算
Fill633.6%动态索引生成
其他25314.4%

8.2.2. INT8 计算层精度分析

虽然 Summary 显示 → FP32: 336 (68.3%),但 Reformat 详情揭示了真相:

FP32 ──→ Row major Int8 format  (量化)
        ↓
   [INT8 卷积计算]
        ↓
Row major Int8 format ──→ Row major linear FP32  (反量化)

INT8 引擎的 Q/DQ 模式导致每个 INT8 计算层前后各有一个 Reformat,这解释了 144 个 Reformat 的来源。计算层本身的输出格式是 FP32(反量化后),所以脚本统计为 FP32,但实际计算确实是 INT8。

8.2.3. INT8 额外问题

  • 193 个 Constant:FP32 只有 4 个。INT8 需要把权重从 FP32 量化到 INT8,每个量化权重块都变成了一个独立的 Constant 层
  • 247 个 PointWiseV2:FP32 只有 19 个。大量逐元素操作来自量化/反量化链中的 scale/zero_point 运算
  • 556 个 NoOp:与 FP16 相同的根源——Reshape/Unsqueeze 显式化,但 INT8 的 Q/DQ 链进一步放大了这个数字

8.3. 根因

8.3.1. 动态shape泛滥

证据:

  • Slice 从 2 膨胀到 156(所有引擎)
  • Fill 从 0 膨胀到 63(所有引擎)
  • Tile_302 的 Size: [-25,4,8192,3],Mode=WRAP,使用隐式 padding mask

原因:ONNX 模型使用了动态维度(如 batch 可变、序列长度可变),TRT 必须为每个动态维插入 Slice/Fill 层来处理 padding 和索引生成。

8.3.2. Reshape/Unsqueeze 过度显式化

证据:

  • NoOp 从 5 → 104 → 556
  • Unsqueeze 相关的 reformat 和 copy 节点大量出现

原因:ONNX 导出时未做 shape 推理和常量折叠,每个 Reshape/Unsqueeze 都被保留为独立节点。

8.3.3. Myelin 融合回退(FP16 特有)

证据:ForeignNode[Add_2568…Unsqueeze_2943] 不支持 FP16,导致 9 层回退。

原因:Myelin 自动融合了 Add + Unsqueeze,但该融合算子未注册 FP16 kernel。

8.4. 修复

8.4.1. 导出ONNX时消除动态Shape

预期效果:Slice 从 156 → ~10,Fill 从 63 → 0,INT8 的 NoOp 减少 30%+

8.4.2. 消除Myelin FP16回退

# 在 trtexec 构建时,禁用 Myelin 或限制其融合范围
trtexec --onnx=model.onnx \
        --fp16 \
        --tacticSources=-myelin \     # 禁用 Myelin tactic
        --saveEngine=fp16.engine

预期效果:消除 9 个 FP32 回退 + 5 个 reformat

状态:8.4.1 与 8.4.2 为待执行项;下面 4 条(P1/P2)仅作记录,暂未执行。

🟠 P1 — 重要优化

3. ONNX 简化(必做)

# 安装 onnx-simplifier
pip install onnx-simplifier

# 简化 ONNX 模型(常量折叠 + shape 推理 + 消除冗余节点)
python -m onnxsim model.onnx model_sim.onnx \
    --dynamic-input-shape \
    --input-shape "input:1,3,800,800"

预期效果:NoOp 减少 50%+,Reshape/Unsqueeze 被折叠,Slice 减少

4. 使用 onnx-graphsurgeon 清理冗余节点

import onnx_graphsurgeon as gs

graph = gs.import_onnx(onnx.load("model_sim.onnx"))

# 消除连续的 NoOp 节点
for node in graph.nodes:
    if node.op == "Identity":
        node.outputs[0].inputs.clear()
        node.inputs[0].outputs = node.outputs
        node.inputs.clear()
        node.outputs.clear()

graph.cleanup()
onnx.save(gs.export_onnx(graph), "model_clean.onnx")

🟡 P2 — INT8 量化改进

5. 使用 PTQ 显式量化(替代 TRT 隐式量化)

# 使用 TensorRT Model Optimizer 做显式量化
# 这比 TRT 自动 INT8 校准产生的 Q/DQ 节点更少、更可控
from tensorrt_llm.quantization import quantize

# 或使用 PyTorch Quantization
import torch.quantization as quant

model.qconfig = quant.get_default_qconfig('fbgemm')
model = quant.prepare_qat(model, inplace=True)
# ... 训练/校准 ...
model = quant.convert(model, inplace=True)

原因:TRT 隐式量化(–int8)会为每个张量独立插入 Q/DQ,导致 144 个 Reformat + 247 个 PointWiseV2。显式量化可以手动控制 Q/DQ 位置,减少 60% 以上的转换节点。

6. 如果必须用 TRT 隐式 INT8

trtexec --onnx=model_sim.onnx \
        --int8 \
        --calib=calibration.cache \
        --layerPrecisions="*:int8" \      # 强制所有层 INT8
        --layerOutputTypes="*:fp32" \     # 统一输出 FP32
        --tacticSources=-myelin \         # 禁用 Myelin
        --saveEngine=int8.engine

附录

TensorRT Layer Info Analysis

Summary Table

MetricFP32 EngineFP16 EngineINT8 Engine
Total Layers994881753
FP3299 (100.0%)9 (3.0%)1051 (100.0%)
FP160 (0.0%)289 (97.0%)0 (0.0%)
N/A0 (0.0%)190 (0.0%)702 (0.0%)
Tactic Nonzero80 (80.8%)105 (21.5%)658 (37.5%)
Tactic Zero19 (19.2%)383 (78.5%)1095 (62.5%)
Reformat Layers5 (5.1%)23 (4.7%)144 (8.2%)
Compute-Heavy Layers82 (82.8%)116 (23.8%)492 (28.1%)
→ FP3282 (100.0%)1 (0.9%)336 (68.3%)
→ FP160 (0.0%)115 (99.1%)0 (0.0%)
FP32 Fallback090

Layer Type Distribution

LayerTypeFP32 EngineFP16 EngineINT8 Engine
NoOp5104556
PointWiseV21922247
Constant414193
Slice2156156
Reformat523144
CaskConvolution5555141
Reduce0280
Fill06363
Shuffle0748
ElementWise01140
CaskGemmConvolution1532
Myelin72223
CaskGemmMatrixMultiply0011
CublasConvolution019
CudaSoftMax006
CaskPooling010
Scale011
ShapeHostToDevice011
CaskConvActPool001
CudnnPooling100
MatrixMultiply001

Reformat Details: FP32 Engine

Total: 5

  • Unsqueeze_3427_copy_input: Row major linear FP32 → Row major linear FP32
  • Reformatting CopyNode for Input Tensor 0 to reshape_after_MatMul_3454: Channel major FP32 format where channel % 4 == 0 → Row major linearFP32
  • 4964 copy: Row major linear FP32 → Row major linear FP32
  • Reformatting CopyNode for Output Tensor 0 to MaxPool_28: Row major linear FP32 → Channel major FP32 format where channel % 4 == 0
  • Reformatting CopyNode for Input Tensor 2 to {ForeignNode[5239...Slice_4578]}: Channel major FP32 format where channel % 4 == 0 → Row majorlinear FP32

Reformat Details: FP16 Engine

Total: 23

  • Slice_324: Row major linear FP16 format → Row major linear FP16 format
  • Unsqueeze_800_copy_output: Row major linear FP16 format → Row major linear FP16 format
  • 1645 copy: Row major linear FP16 format → Row major linear FP16 format
  • Slice_830: Row major linear FP16 format → Row major linear FP16 format
  • Reformatting CopyNode for Input Tensor 0 to Conv_26 + Relu_27: Row major linear FP32 → Channel major FP16 format where channel % 4 == 0
  • Reformatting CopyNode for Input Tensor 3 to {ForeignNode[1523...Reshape_1690]}: Channel major FP16 format where channel % 8 == 0 → Row major linear FP16 format
  • Slice_1707: Row major linear FP16 format → Row major linear FP16 format
  • Slice_2584: Row major linear FP16 format → Row major linear FP32
  • Reformatting CopyNode for Input Tensor 0 to {ForeignNode[Add_2568...Unsqueeze_2943]}: Row major linear FP16 format → Row major linear FP32
  • Reformatting CopyNode for Input Tensor 1 to {ForeignNode[Add_2568...Unsqueeze_2943]}: Row major linear FP16 format → Row major linear FP32
  • Reformatting CopyNode for Input Tensor 2 to {ForeignNode[Add_2568...Unsqueeze_2943]}: Row major linear FP16 format → Row major linear FP32
  • Reformatting CopyNode for Input Tensor 3 to {ForeignNode[Add_2568...Unsqueeze_2943]}: Row major linear FP16 format → Row major linear FP32
  • Unsqueeze_3427_copy_input: Row major linear FP16 format → Row major linear FP16 format
  • 4964 copy: Row major linear FP16 format → Row major linear FP16 format
  • Unsqueeze_3441_copy_input: Row major linear FP16 format → Row major linear FP16 format
  • Reformatting CopyNode for Input Tensor 0 to Expand_2957: Row major linear FP32 → Row major linear FP16 format
  • Reformatting CopyNode for Input Tensor 6 to {ForeignNode[5239...Add_4052]}: Row major linear FP16 format → Channel major FP16 format wherechannel % 8 == 0
  • Reformatting CopyNode for Input Tensor 7 to {ForeignNode[5239...Add_4052]}: Row major linear FP16 format → Channel major FP16 format wherechannel % 8 == 0
  • Reformatting CopyNode for Input Tensor 8 to {ForeignNode[5239...Add_4052]}: Row major linear FP16 format → Channel major FP16 format wherechannel % 8 == 0
  • Reformatting CopyNode for Input Tensor 9 to {ForeignNode[5239...Add_4052]}: Row major linear FP16 format → Channel major FP16 format wherechannel % 8 == 0
  • … (3 more)

Reformat Details: INT8 Engine

Total: 144

  • Slice_324: Row major linear FP32 → Row major linear FP32
  • Unsqueeze_800_copy_output: Row major linear FP32 → Row major Int8 format
  • 1645 copy: Row major Int8 format → Row major Int8 format
  • Slice_830: Row major Int8 format → Row major linear FP32
  • Reformatting CopyNode for Output Tensor 0 to {ForeignNode[model.pts_bbox_head.positional_encoding.col_embed.weight...Add_536]}: Row major linear FP32 → Row major Int8 format
  • Reformatting CopyNode for Input Tensor 0 to reshape_before_MatMul_850: Row major linear FP32 → Row major Int8 format
  • reshape_after_MatMul_850_copy_input: Row major Int8 format → Row major linear FP32
  • reshape_after_MatMul_842_copy_input: Row major Int8 format → Row major linear FP32
  • Reformatting CopyNode for Input Tensor 0 to Conv_26 + Relu_27 + MaxPool_28: Row major linear FP32 → Row major Int8 format
  • Reformatting CopyNode for Input Tensor 0 to Transpose_1210 + Reshape_1220 + reshape_before_MatMul_1250: Row major linear FP32 → Row major Int8 format
  • reshape_after_MatMul_1250_copy_input: Row major Int8 format → Row major linear FP32
  • Reformatting CopyNode for Input Tensor 0 to Reshape_1230 + reshape_before_MatMul_1266: Row major linear FP32 → Row major Int8 format
  • reshape_after_MatMul_1266_copy_input: Row major Int8 format → Row major linear FP32
  • reshape_after_MatMul_1258_copy_input: Row major Int8 format → Row major linear FP32
  • Reformatting CopyNode for Output Tensor 0 to PWN(PWN(PWN(PWN(PWN(2702 + (Unnamed Layer* 2710) [Shuffle], Add_1645), Sqrt_1646), Div_1647), Mul_1648), Add_1649): Row major linear FP32 → Row major Int8 format
  • Reformatting CopyNode for Output Tensor 0 to model.pts_bbox_head.transformer.encoder.layers.0.ffns.0.layers.0.0.bias + (Unnamed Layer* 2724) [Shuffle]: Row major linear FP32 → Row major Int8 format
  • Reformatting CopyNode for Input Tensor 0 to PWN(Add_1654, Add_1655): Row major linear FP32 → Row major Int8 format
  • Reformatting CopyNode for Output Tensor 0 to PWN(Add_1654, Add_1655): Row major Int8 format → Row major linear FP32
  • Reformatting CopyNode for Output Tensor 0 to PWN(PWN(PWN(PWN(PWN(2721 + (Unnamed Layer* 2741) [Shuffle], Add_1662), Sqrt_1663), Div_1664), Mul_1665), Add_1666): Row major linear FP32 → Row major Int8 format
  • Unsqueeze_1677_copy_output: Row major Int8 format → Row major Int8 format
  • … (124 more)

FP32 Fallback: FP16 Engine

Total fallback layers: 9

  • [Reformat] x7: Slice_2584, Reformatting CopyNode for Input Tensor 0 to {ForeignNode[Add_2568...Unsqueeze_2943]}, Reformatting CopyNode for Input Tensor 1 to {ForeignNode[Add_2568...Unsqueeze_2943]}, Reformatting CopyNode for Input Tensor 2 to {ForeignNode[Add_2568...Unsqueeze_2943]}, Reformatting CopyNode for Input Tensor 3 to {ForeignNode[Add_2568...Unsqueeze_2943]} … (2 more)
  • [NoOp] x1: Reshape_25
  • [Myelin] x1: {ForeignNode[Add_2568...Unsqueeze_2943]}