I ride with my windows down all of the time. Pretty much daily on my evening commute. I don't have buffeting like you are describing. On a side, note, this was an interesting find - particularly the top comment.
https://www.reddit.com/r/askscience/...e_window_down/