In Shulchan Orach Siman 124 Sif 7 the machaber speaks about how bad it is to speak during Chazaras HaShatz. The Shulchan Aruch HaRav over there in Siman 124 in Sif 10 writes that someone who talks in a shul when the time the tzibbur is “oskin b’shivcho shel makom” that he makes himself as if he doesn’t have “chelek” in Elokei Yisrael. His makor is from the Zohar in Parshas Trumah (Daf 131b.) As well the Shulchan Aurch in Orach Chaim Siman 151 Sif 1 writes that “sicha butteleh” is ossur to speak in a Beis HaKeneses or Beis HaMedresh. This is talking about in general, how much more so when the Tzibbur is davening from what we see above. See too what the M”B writes there in Sif-Katan 2.
With this small introduction my question is as follows. Does it say any where b’farush that it’s “ossur” to speak in the middle of Kaddish. Let’s say it’s a person who is not davening with the tzibbur at all who comes into the shul for a different reason. Or someone who plans on davening with the tzibbur but hasn’t started yet and is l’maseh able to speak still but is speaking now during Kaddish. Where does it say that this is ossur or where do the poskim learn it out from? The makoros that I provided above would seemingly suffice I think but I’m looking to see where there is more.
See also Eliyah Rabbah 124 citing from Kol Bo.
Even somebody who is not davening must not speak in the middle of Kaddish while in the shul, for this is a zilzul (demonstrates a lack of respect) for Kaddish.
I don’t know of explicit sources on this, but if it is forbidden to speak during Chazaras Ha-Shatz, it is the more so forbidden to speak in Kaddish.
An interesting anecdotal source is the Brisk Rav, who used to be careful not to answer Amen after ve-yatzmach purkanei (at least in the minyan in his own home), explaining that this is a hefsek in Kaddish.