Question: How long should saving the checkpoint take?

[dvkeulen]

When the checkpoint is saved while training, how long should this process take? It seems to me that my program doesn't run at the point of saving.

In the image I set the checkpoint saving threshold to 10 for testing purposes.

[dvkeulen]

edit: it does finish it, but it takes 2 whole hours to save the checkpoints. Is this normal behaviour?

[YangZeyu95]

That is preparing the test data.