I think the problem with the handlebars is that the lip or tongue of that special spacer is contacting the handlebars. I have seen some markings where it contacts the handlebar and marked them in green. I guess the simple solution is trying to file this spacer down until it sits flush.

