I originally posted a thread that didn't get any responses but was quite wordy. Don't even bother reading it:
I did a lot more testing and have narrowed down the problem. Maybe someone can help me understand what is going on.
I have a 32" TV that can output the "raw" audio signal that is sent to it via HDMI through a coax output. It can also decode a 5.1 Dolby Digital signal and downconvert it to 2 channel PCM. When I set the PS3 HDMI output to automatically check the TV's audio settings via a handshake it returns:
Dolby Digital 5.1
Linear PCM 44.1 khz
Linear PCM 48 khz
If I stream or transcode any .mkv or .m2ts files with an AC3 audio track; the amp will receive a 2 channel PCM input if "PCM" is selected on the TV's digital audio out because the TV has down converted it. The amp will receive an untouched Dolby Digital 5.1 channel input if I select "RAW" on the TV's digital audio out.
This is working as expected. The problem is when trying to pass the DTS track in the LPCM header. I can hit the "display" button and see that the DTS audio track is recognized by the PS3 as Linear PCM on the screen however the amp output is static.
Presumably the PS3 is doing something to the track before getting it to the TV. There is no way to force bitstreaming other than physical media (BD/DVD).
I know from experimenting with DVDs that if you set the PS3 to bitstream and manually set all the HDMI audio formats on; the PS3 will not output audio if the TV connected to it doesn't support the audio track you are trying to force over. The PS3 knows what the TV can handle via the initial handshake and refuses to send over unsupported formats. No static or damage to your TV.
In my case the LPCM audio track is being modified before getting to the amp. There is no "bitstream" option outside of physical media. I believe the PS3 is modifying the signal before it gets sent to the TV. Presumably something in the header doesn't jive with the results it got back from the handshake so it does some type of conversion that destroys the DTS track. Any thoughts?