Gimp transparent layer background

In Photoshop when I remove an area from a layer by selecting the area and pressing the delete button, then the removed area becomes transparent.

By Gimp the area gets the color of the background (white). Is there a way to have a photoshop like behavior in Gimp or make parts of a layer transparent?

Another solution would be for my problem to copy an area into a new layer with transparent background, but that does not work out of the box either. :S