Question: how to adapt this for CTC loss

[pfeatherstone]

@lucidrains Do you have any advice on how to adapt RecurrentMemoryTransformerWrapper such that it works with CTC ?

[pfeatherstone]

In the memory replay backpropagation algorithm, the labels are partitioned in the same way as the logits. The loss is evaluated per block. For CTC that doesn't make sense since labels are not necessarily aligned.... So does memory replay in its current form even apply to CTC?? Any help is gratefully received.

[pfeatherstone]

@lucidrains Or if we forget CTC, can you think of a way to make this work with unaligned targets ?