what teshuva can one do if one mistakenly miscalculated and had relations on the onah beinonos
Firstly, to ensure proper care is being taken that it does not happen again. To make sure the system of calculating is clear and accurate, that it is checked and double checked by husband and wife.
In addition many recommend that the couple learn the pertinent halachos of vestos in specific, and of nidda in general. Reading a sefer of the halachos, attending a refresher course or something similar.