I wonder if part of his meaning is that, when you're not physically in the office, there are things you're missing out on that make you more efficient. You miss conversations and context. I know for a fact that I've advanced more quickly in my role than some others who joined during COVID because of the months pre-COVID when I worked with colleagues in the office. I learned who everyone was, who had what information, what everyone felt was important to the business, and built strong relationships. It made me 10x more effective at my job.
So even though everyone works the same number of hours, I suspect that the people who spend time in the office are able to get more done.