Most likely the audio is either E-AC3 which i dont think PMS supports, or the video codec is VC-1, which about 90% of HD DVD movies are. PS3 does not support VC-1 streamed, period. Any movie on VC-1 on both formats, just grab a re-encode of it, there is no way around this. Also, even HD DVD movies in AVC have issues streaming, they are encoded with 29.97 instead of 23.97 like Blu-rays are and PMS does not like them at all. Just a heads up. When it comes to anything HD DVD, just grab a re-encode and forget that format ever existed like the rest of the world already did.
p.s. - If you want to transcode the VC-1 into mpeg2 with mencoder, you can, but you will need the vc-1 wmv dll in the PMS codec folder. Do a search, the url got posted earlier. My comment above was strictly for streaming without transcoding, which is always the best way to handle streaming to the ps3.