Personally, I tend to scout it and flank it.
Example, Use the Officer with his Binocs or get a soldier close enough to find out where its facing, Either move an AT gun or Tank to a flank, Track it so it can't face you. Blow its weak armor apart.
The biggest weakness for Tank Destroyers is that when tracked they effectively become useless. Not speaking for the US Or Commonwealth units. They just have weak armor overall so hit them before they can hit you kinda thing.
TDs are normally placed defensively due to limited sight and mobility so you can usually poke around until you find out where they are.
This is my opinion at least, I'm sure there are more effective ways but it works for me 8 out of 10 times.