I searched a little while, found only AS2 workarounds, but here is the AS3 version of interpolated, smooth video scaling of a FLVPlayback.

//playback is your FLVplayback instance
playback.getVideoPlayer(playback.activeVideoPlayerIndex).smoothing = true;

I use it after the ready event was send.