You have options when you're setting up voicemail. Therefore, you're in control. You can either:
1. Not set up voicemail -- This lets everyone know that you don't communicate via voicemail.
2. Set up voicemail, with a message that says something like "I don't check messages often, so please send me a text or email if it's urgent." -- This lets everyone know that voicemails are for routine matters and anything with a deadline should go through another medium.
3. Set up voicemail, with the usual message of "This is X, please leave a message!" -- This lets everyone know that it's acceptable to communicate with you via voicemail.
If you choose Option 3, then you are rude if you're expecting people to contact you in a different way for urgent matters. You're telling them one thing and then expecting them to read your mind and do something different. It's easy enough to take one of the other options and clearly let everyone know what to expect. (There's also the option of "Don't leave a message, I'll just call you back when I see the missed call." I hate leaving a detailed message as instructed and then having someone immediately call me back without listening to it and making me repeat everything.)