[cleanspeed1]

It comes down, to a great part, to the chemistry of the fuel that is being used.

Increasing the compression ratio, which increases the pressure and heat inside the cylinder creates, for lack of a better term, a bigger "bang" that should allow for more complete combustion and less loss in terms of radiated heat.

Gasoline direct injection allows higher compression ratios, but once again, if the fuel quality is such that it does not react completely, you will have pre-ignition/detonation problems. This has been dealt with through higher injection pressures, better atomization and knock sensing ignition systems by and large.

Every fuel has an ideal fuel to air ratio chemically and diesel fuel is no exception.

It's all about control. Diesel engines are designed to withstand high cylinder pressures and abnormal combustion events, along with very lean mixtures. The fuel allows this.
That rattle that comes from a diesel? Detonation / pre ignition.

As long as the fuel / air mix is prepared properly before introduction into the combustion chamber for reaction, and the harder you compress it, the greater and more thorough the release on the power stroke.

Gases burn completely, vapors burn completely, droplets do not. Because most fuel systems take things down to droplet size, whether carburetor or fuel injection, combustion is never complete and hence aftertreatment is needed to deal with emissions.
If the reaction was complete inside the engine, then it would not produce emissions and the fuel economy would go up. The engine would run cooler.

If I missed something, I apologize in advance; there is so much involved.