I read in halacha that if one forgets on motzei shabbos to recite havdala both in maariv and on wine and remembered only after eating then he is obligated to recite both maariv again and havdala,as a fine for forgetting. Does that apply on motzei shabbos that is a yom tov, if he forgot to recite both vatodianu and havdala during the kiddush?
Yes. The halachah is that if a person forgets atah chonantanu and then eats before havdalah, he must daven again.
This applies to va-todienu in the same way, since this is the havdalah when Sunday is a Yom Tov (Biur Halachah 294).