MKVs, and the Xbox 360....

[thel0be]

edited

[hursto75]

this may sound stupid but change the mkv extension to m4v and try it again.

[patters]

When you say it's looking pixelated when it plays back on the 360, that's probably because PS3MS is using mencoder to transcode it to MPEG2 which will look worse. Try bumping tsMuxer up the list to be the preferred encoder.
tsMuxer is able to unwrap the encoded audio and video from the MKV file (which the 360 cannot play) and, provided the H.264 has been encoded to right profile to be playable on the 360 it should be able to play it without transcoding. You could have additional problems with audio though since AFAIK the 360 can only receive stereo audio as a DLNA renderer. Though I have a 360, I've been using my Bravia TV for PS3MS instead. A couple of quick tries on my 360 were really stuttery and choppy.