Not hugely - it will still work and it will be hidden from view. But it would bother me - I usually get a sheet sander out after I've cut my steerer tube and even it all out which would fix that really quickly (and if you don't own one, they are a really useful DIY tool and in the UK cost from about £10, so not much). Make sure you do it outside and wear a mask as you would when cutting it.