I checked the Disney dining guide and this is what I found:
Men: Dress slacks, jeans, trousers, or dress shorts, short- or long-sleeved shirt with a collar or t-shirt required. Jackets are optional.
Ladies: Jeans, skirt, or dress shorts with blouse, sweater or t-shirt, or a dress required.
Not permitted: tank tops, swimsuits or swimsuit cover-ups, hats for gentlemen, cut offs, torn clothing of any kind, or t-shirts with
offensive language and/or graphics.
So, I think nice jeans or khakis will be fine. I have a few options as far as tops and sweaters to coordinate.