It's been a few years since I was up that way.
Port Douglas is beautiful, but when I was there, it was very 'high end tourist' - expensive resorts, boutique shops etc. Cairns has a wider range of options, but it is still Tropical Tourist Town. There have been a couple of cyclones through there since, so things may have changed.
If you're hiring a car, be careful where you park it overnight, my car was vandalised one night there, in the hotel car park.
There's heaps to see and do if you don't want to just do the water things.
Paronella Park is an easy drive from Cairns, it's a Spanish style building in a bit of rainforest handbuilt by a Spanish immigrant in the 1930's for his bride. The builder was very innovative - it's built of mud, railway sleepers and kerosene tins, and he handbuilt a hydroelectric system so he had the first electricity in the area. It is stunningly beautiful and worthwhile visiting - if you're lucky you may fluke a time when there's a Shakespeare production on there.
Kuranda is an easy drive up the mountain, it's a bit touristy too, but interesting. Apparently the markets are great, but I missed them. Lady Barron Falls is spectacular when there's been any rain.
There's also white water rafting, for a day or for several days, depending on the tour you want.
The Atherton Tablelands make for an interesting day drive, the scenery is spectacular.