I'd recommend just lowering your pressure bit by bit to see how it feels. You should feel the increase in rolling resistance when you go too low, but before that, there should be a noticable increase in comfort.
For what it's worth, I run 45-60psi in my 28mm road tires, and 30-35psi in wider gravel tires (tubeless in both cases). But I live in Belgium, so very little smooth tarmac here.