I think you need to approach this in a business fashion, not a friend fashion.
Send him a polite but clear letter, telling him that you will be taking up a new job, and therefore will be cancelling your prior arrangement with him. As of X date, you will not longer be available for work. Give appropriate notice, so he can find someone else.
How he responds to this will go a long way in telling you if there is a chance of becoming friends with him again. If he throws a hissy fit, refuses to take no for an answer, or keeps guilt tripping your or harassing you over it, then you know that there isn't much of a friendship left. Back off, and let him make an overture if he want to when he calms down.
Make sure to refuse *completely* if he makes any more work related requests, even if it's a minor favor. You've set a precedent of letting him take advantage of you, both as a friend and as an employee, so it's going to take work to break that habit. He's got a pretty sweet deal right now - a personal assistant, on call, whom he can stiff for salary and expenses. He's unlikely to find something as good, so it's in his interest to push for it.