I do just about the same thing, and by doing this and switching back, you not only prevent the air from getting stale but also prevent the "frostbite" feeling that occurs once the car IS cooler down. It lets you leave the ac at 2 or 3 and have a breeze without freezing to death. 15 minutes earlier, on these hot days, it seems like you'd have given anything to freeze to death and thus the recirculate option at first.Correct, and there's a reason for this. If you set the control to recirculate and the compressor didn't run, the air in the car would become stale very quickly and you would get quite a bit of fogging. Without the compressor running, you need to draw in outside fresh air to minimize this. In recirculate, the air can become stale and stuffy even with the compressor running. Without the compressor, it would be magnified much more.
I only use recirculate to avoid outside odors (exhaust fumes from a bus) or to cool the car quickly on a very hot day. You will notice that the air is colder when you run the A/C in recirculate mode. I always switch to outside air soon after.
Why does it become stale and stuffy when recirculating less with the compressor though? I don't follow that but did know that it comes on, of course.