Smart-casual means one tailored element and one relaxed one. A blazer with denim. A knit with wool trousers. Chinos with a proper shoe.
The mistake is going halfway on everything — that's how you end up looking neither. Commit to one sharp piece and let the rest sit back.