It's both of course.
The currency of weight is certainly calories, and while we all have our own unique internal fuel efficiencies when it comes to using or extracting energy from food or from our fat stores, we still need a surplus of calories to gain and a deficit to lose.
But foods matter too. Choice of food matters in terms of health, but also in terms of how many calories our body expends in digesting, and more importantly, upon satiety, which in turn has a marked impact upon how many calories, and which foods, we choose to eat.
So if you do come across a zealot from either camp that claims one or the other doesn't matter, feel free to ignore them.