Is there a Halakha in which say that the Challah in Rosh Hashanah has to be round?
It is not an obligation, however many have such a minhag. Some of the reasons for the minhag are that it is made round to symbolize the circle of the life and the cycle of a new year.
The challah is also in the shape of a crown, indicating the coronation of Hashem on Rosh Hashanah, which is the central theme of the day.
Piskei Teshuvos 583 ftnt. 46