I do just about the same thing, and by doing this and switching back, you not only prevent the air from getting stale but also prevent the "frostbite" feeling that occurs once the car IS cooler down. It lets you leave the ac at 2 or 3 and have a breeze without freezing to death. 15 minutes earlier, on these hot days, it seems like you'd have given anything to freeze to death and thus the recirculate option at first.
Why does it become stale and stuffy when recirculating less with the compressor though? I don't follow that but did know that it comes on, of course.