The MB (307,16) laments how people waste their time by going to circuses and the theater, and are moshav leitzim. What would be the approach to today’s concerts – either with a frum performer or an orchestra just playing classical music? And what about an amusement park that has rides? Would it depend on what stature the person is, why he is going (to relieve stress or spend time with his family etc.)?
It says “sh’alas chochom chatzei teshuva”, a smart person’s question is half the answer. You basically included the answer in your question.
Yes it is true that we should not waste time amusing ourselves, but we should spend our time learning, doing mitzvos and chesed. However each person has to know what they have to do, and when. As you mentioned, there are times when it might be needed, such as if it is needed to relieve stress or to spend time with his family, each person according to his needs.