Working with literally thousands of patients over the course of the past dozen years I can tell you people don't gain weight consequent to a slow, linear, one extra potato chip too many per day, sort of way. People gain weight consequent to big, giant, clumps of calories.
Sometimes those clumps come from restaurant meals. Sometimes those clumps come from vacations. Sometimes they come from religious holidays and family gatherings. Sometimes they come from comfort eating after an especially rough week. Sometimes they come from a weekend or a night out with the girls (or boys). Sometimes they come from burning the midnight oil. No doubt for most of us they come on our, and our kids' birthdays.
And none of this is to say that the quality of calories don't matter. They do, as the quality of our calories will directly and indirectly affect the size of our clumps. But there is no shortage of calorie clumps.
Even if these clumps when squished together made up only 5% of our years' days, and assuming each of those days provided only a conservative 500 calorie clump (I'm betting many day's clumps climb easily into the thousands), those clumps' contributions to calories could more than explain our slow, annual, average rise in weight - even if the remaining 95% of our days were in perfect caloric harmony.