I often refer to someone who maybe in the same room as me as "he" or "she" rather than saying their name. It is not impolite, if everyone knows who you are talking about, it is just quicker.
The way they are doing it is definitely impolite, they're having a conversation about him as though he's not there. It's not instantly rude to refer to someone as he or she but the way they're doing it definitely is.