BSFC charts show that an engine under high load runs more efficient than an engine at low load. This is why smaller engines work more efficiently. You can not 'work an engine too hard'. If we look at a BSFC chart you can see why.
The top line is peak torque output. That is what you get when you are at WOT (wide open throttle). Along that line, worst case scenario in the above map is at 6000 rpm (cause nobody drives around at 750 rpm). So, even if you are beating the living snot out of your engine at WOT and peak rpms, you are likely still more efficient than normal cruising, and especially so with a larger engine.
For example, in this case lets say the car is crusing at 65 mph and uses 14 hp to do that (I went through this example a while back
here). The red points are where the engine would operate at different rpms (different gearing). If we had a smaller engine, all those points would be shifted upwards which means higher engine loads. That also means the engine would operate more efficiently. A larger engine would shift them all downwards resulting in the opposite.
You can see it is theoretically possible to operate the engine at such high loads that you SLIGHTLY miss peak efficiency. This means driving around at WOT ALL THE TIME. This simply does not happen.