I searched a little while, found only AS2 workarounds, but here is the AS3 version of interpolated, smooth video scaling of a FLVPlayback.
playback.getVideoPlayer(playback.activeVideoPlayerIndex).smoothing = true;
I use it after the ready event was send.