PS3 supports m2ts files natively so you can play them directly. Playlist is not supported but PS3 has settings to continue after one file is played so you can play all files in one directory continually (with some pauses between each file I suppose)
So my question is can I play MKV's X264 without transcoding as long as it contains MPEG-4 AVC video?
Yes, not natively but remuxed by txMuxeR. MKV container is changed to m2ts on the fly without any loss of quality but you will loose subtitle support (PS3 can't handle it)