To see what messages are being exchanged between the DLNA server and renderer you could use wireshark to reverse engineer the protocol (quite a hassle, but as you said, the DLNA spec isn't open and thus costly).
You can have a look at the net.pms.network.RequestV2 class. I haven't laid hands on this part of the code (and won't as its working fine), but it seems to be the one handling the DLNA communication. I guess extracting it won't be that easy, as its quite tightly linked to the rest of the implementation.
If you always wanted to have your most recent videos at the top of the folder in the ps3 or wished you could group all videos with the same genre in one folder, have a look at pms-mlx