Oddly enough, I've had to adjust my settings some with the new PS3 update. I updated a few days ago and not a single one of my 1080p movies would stream without stuttering again. After about half an hour of trial and error, I now have it streaming smoothly again. The adjustments I've had to make are as follows:
Maximum bandwidth changed from 0 to 40. No idea why I had to do this now when 0 was okay before, but I did. I'm using a powerline cable set up, so I don't have to have a 75ft cable going from my computer room to the living room.
I'd highly recommend doing that over trying to use wireless for 1080p.
Quality settings is now set to keyint=25:vqscale=1:vqmin=2 Again, no idea why changing vqmin from 1 to 2 helped when it was fine at 1 before.
And now the real kicker. I've had to make TsMuxer my primary video engine in order to stream nearly flawlessly instead of Avisynth/MEncoder. Don't know why again. It works best with that now. It does help to allow the buffer to fill up completely before starting the movie too. I even went into the Transcode folder and tried every single selection available. TsMuxer still worked best with almost every single video. There was one Blu-Ray rip of Zohan that I had various rips of for testing purposes. The mpg rip worked better with MEncoder. The ONLY change between my system and the PS3 was the PS3 update too. I haven't changed how I rip my DVD's and Blu-Ray discs from my collection. I haven't changed any codecs. Nada. Pretty frickin odd!
Also, while messing around with settings to get things smooth again, I dropped the K-lite pack I was using and went to CCCP as well as updating TsMuxer in the PMS Win32 folder. Doesn't seem to have made any difference, but I wanted to be sure.
I'd also like to say if you have the right hardware to take advantage of CoreAVC, go buy it! It's cheap and takes quite a bit of strain off your processor if you need to. I've been using it for about 6 months now.