Actually, there are times when I will take a seat that vacates near me simply BECAUSE it moves me OUT of the aisle, which means people don't have to maneuver past me to get out.
So in a way, if there's no one immediately near you who needs the seat, I think you *should* sit down, if only to get out of the walking space.