The Mishna Berura(Shulchan Aruch Siman 188 Sif Katan 32) brings down that if one would daven maariv motzei shabbos before bentching he should not bentch with retzai anymore. In another place(Shulchan Aruch 263 Sif Katan 67) the Mishna Berura leaves it as tzarich eeyin if one would say “hamavdil bein kodesh lchol” before bentching motzei shabbos if he should continue bentching with retzai or not.
My first qstn is lmaisa wut should one do if they do say “hamvdil bein kodesh lchol”?
My second qstn is what would the halacha be if one said havdalah before bentching. Does it fall into the same “tzarich eeyin” as saying “hamavdil beiin kodesh lchol” or would it be like davening maariv?
After davening maariv (even if atah chonantanu was not recited) one may not continue to recite retzei. Although Mishnah Berurah leaves it as tzarich iyun, it would seem that the same is true of one who said baruch hamavdil whena prohibited act of labor was performed. If not act of labor is performed, it remains a safek, and retezei should not be said. Assuming there is a difference between the two, reciting havdalah, which includes the blessing of hamavdil bein kodesh lechol, would be equivalent to davening maariv.
Sources: Mishnah Berurah (188:32); Birkas Habayis (shaar 17) explains that this (one who davens maariv) applies even when atah chonantanu was not said, because maariv itself is sufficient to prohibit further eating before havdalah; see also Birur Halachah (299:3). Yet, see Har Ha-Carmel (7), who writes that the problem of reciting retzei is on account of the prior recitation of atah chonantanu (he derives this from Maharil, but in my humble opinion, the Maharil (cited in Magen Avraham 188:17) who only mentions “the weekday prayer,” does not mean to infer the recitation of atah chonantanu, but only the weekday maariv prayer).
Concerning the recitation of baruch hamavdil, Mishnah Berurah (623:67) mentions the halachah as a safek, which is already mentioned in Tosafos Shabbos. A number of authorities mandate reciting retzei under such circumstances (Eliyah Rabbah 299:1, Kitzur Shelah 1:47 and Kaf Ha-Chaim 27), whereas others write that it should not be recited (Shulchan Aruch Ha-Graz 188:17, Me’orei Or 263). Out of doubt, because the ommission of retzei does not invalidate the blessing, retzei should not be recited, the more so if nightfall has already arrived, when the recitation of retzei would be a contradiction to the permission to perform acts of labor (Birur Halachah 299:4) — and certainly when an act of labor was actually performed, which would appear to preclude a person from reciting retzei no less than davening maariv.
Reciting havdalah is like davening maariv for all intents and purposes, just as we see concerning the beginning of Shabbos when reciting kiddush has the same halachos as davening maariv (with regard to the acceptance of Shabbos). Just as kiddush is the most forceful means of accepting Shabbos (according to a number of rishonim, kiddush, together with davening maariv, is the sole means of accepting Shabbos early), so reciting havdalah is the most forceful means of bringing out Shabbos. It would therefore be equal to davening maariv with regard to the recitation of retzei.