Just a comment about Chatsworth - it is gorgeous, but it's a private home, so also quite pricey - ~£20 a ticket for an adult + parking on top (unless you reserve online). That said, it's in a beautiful part of the countryside, and is a beautiful house - and it has a playground for the littlies. Nearby Haddon Hall (Tudor Manor House) and Bakewell (mmm cake) worth a visit, as well as a climb up Thorpe Cloud (v. steep bring good shoes) to get a feel for the Peak District. If you want to do a lot of castles/houses though, NT/EH membership worth investigating.
Other suggestions for visit to Somerset/Bath area: Salisbury (beautiful cathedral with lovely views from the water meadows - maybe attend a choral evensong there), Avebury (better than Stonehenge in my opinion, which I think is decidedly underwhelming), Stourhead (National Trust stately home with amazing gardens), and Old Wardour Castle (English Heritage - ruined castle, v. picturesque), also Lacock, which is where they film a lot of period/regency dramas.