If someone by mistake said “Atah yatzarta” by musaf on Shabbos Erev Rosh Chodesh do they need to daven musaf over again?
He must repeat the Shemoneh Esrei.
If he were to say the regular Mussaf prayer, but conclude with the Rosh Chodesh blessing, the Shulchan Aruch rules that he would not have to repeat Mussaf (see Shulchan Aruch 108:12).
This is because he has said the correct prayer, and the fact that he added the words “rosh chodesh” does not detract from the blessing (see rishonim to Berachos 12; Birchei Yosef 490:6).
The Mishnah Berurah (end of 108) adds that many authorities maintain that the additional words are a hefsek, and they are equivalent to speaking in the middle of davening.
However, in this case he has not recited the correct berachah at all, and instead of the Shabbos berachah, he has said the special berachah for Shabbos and Rosh Chodesh. This is a complete change from the matbe’a (liturgy) that Chazal enacted, and he must therefore repeat the Mussaf prayer again.