I don’t think that’s what is happening in my situation. I have a 60/50 head and then KF40 all the way to the pump - I can’t imagine I have any vacuum restrictions. My gauge responds quickly to anything happening in my boiling flask. If I kick up the stir speed, my gauge instantly jumps.
I think the simplest explanation is the right answer. My collection flask is significantly cooler than my condenser, so of course any vapor that doesn’t condense will start to collect at any cold points.