So what caused the end of Malthusian age? How did humanity escape from the trap in that invention and ingenuity increased the numbers though not the material well-being of humanity?

The key is that even in Malthusian age, the pace at which inventions were made increased steadily. First of all, the population grew. Inventions made communication easier: especially after the invention of printing knowledge could diffuse quickly and widely. More people meant more inventions: two heads are greater than one. The rate of technological progress slowly rose over millennia. And about 1500 it passed the point at that natural resource scarcity couldn't fully offset it. Sustained increases not just in population though in the productivity of labor followed.

At first the rise in material standards of living brought sharp increases in the rate of population growth: population explosion. However as material standards of living rose far above subsistence, countries began to undergo the demographic transition.