Bulk usually isn't the sweater's fault, it's the proportion around it. A chunky knit needs a slimmer trouser; a fine-gauge one can take a wider leg.
A half-tuck at the front defines the waist without looking fussy. And if you're layering a shirt underneath, keep the collar flat — a stiff collar under a crewneck is what makes the whole thing look padded.