[tests] fix autoencoderkl dtype tests#14190
Conversation
| # The reference and reloaded models hold identical weights, so any output difference is | ||
| # half-precision kernel nondeterminism between the two module instances rather than a save/load | ||
| # fidelity issue. The default 1e-4 tolerance is too tight for that fp16/bf16 noise on some GPUs. | ||
| super().test_from_save_pretrained_dtype_inference(tmp_path, dtype, atol=1e-3) |
There was a problem hiding this comment.
It looks like even with an atol of 1e-3, the test still fails (https://github.com/huggingface/diffusers/actions/runs/29625018289/job/88027676669?pr=14190):
FAILED tests/models/autoencoders/test_models_autoencoder_kl.py::TestAutoencoderKL::test_from_save_pretrained_dtype_inference[bf16] - AssertionError: Loaded model output differs for torch.bfloat16
Tensors not close! Mismatched elements: 17/12288 (0.1%)
Max diff: 3.906250e-03 at index (2, 1, 3, 13)
Actual: -5.625000e-01
Expected: -5.585938e-01
atol: 1.000000e-03, rtol: 0.000000e+00I chatted with Claude about this and it suggested using a non-zero rtol, for example following the torch.testing.assert_close recommended tolerance values, which are atol=1e-5 and rtol=1.6e-2 for BF16. I think this makes sense as it makes the test more robust to the scale of the outputs (for example, if the outputs are really big, their absolute difference might be greater than 1e-3 even if they are relatively close and the difference is just due to numerical noise).
Claude also suggests per-dtype tolerances, which I also think make sense since the amount of expected noise should vary between FP16 and BF16 (for example, torch.testing.assert_close uses a rtol of 1e-3 for FP16).
There was a problem hiding this comment.
Claude also suggests per-dtype tolerances, which I also think make sense since the amount of expected noise should vary between FP16 and BF16 (for example, torch.testing.assert_close uses a rtol of 1e-3 for FP16).
This is actually a good suggestion but perhaps the scope isn't quite there in this PR.
Rest -- I have addressed.
2 participants
https://github.com/huggingface/diffusers/actions/runs/29302349254/job/87122984094?pr=14162