At this point, I suppose the clincher would be for the idea/script for IM4 to be something really fresh, innovative and challenging. If the money alone won't do it.
I suspect that in terms of both Avengers 2 and other movies, it might be workable for RDJ to do some Tony Stark cameos, or shoot some scenes that wouldn't take a lot of time (on his part, the CGI armor being what takes time) , if he's worried about not having time for other things.
Most likely any hesitation might just be related to negotiating positions, or that momentary "fatigue" actors have with projects after a lot of time on set or after endless, boring and repetitive press junkets.
I'm not really worried about RDJ disappearing from the Marvel Movies. The only real issue is what his involvement will be and what would make the 4th movie worth doing, creatively.