I'm not quite sure if I fully understand your post.
Youtube plugin has quality and container chooser, you can access it via plugin option (Options -> Preferences -> Plugins -> Settings, select YouTube plugin then press Option button).
Preferred quality has higher priority than preferred container.
If the preferred quality doesn't exist the plugin will pick the nearest lower quality. If the preferred container doesn't exist the plugin will pick container base on this priority : mp4 > flv > webm > 3gp.