It has been covered.... several times actually....
Reason: You're using wifi
Answer 1: Wire the connection and don't take a hit in quality
Answer 2: Decrease the maximum bandwidth (start with 15mbps) and decrease the output quality settings (both under the Common Transcode settings)
Side note: you can't trust the bitrate provided by the PS3 when the video is stuttering as it's grossly inflated