On your active Scene Layers click the Trashcan icon next to a the Layer you want to delete and the layer be deleted.

Note: You can delete only those Layers with a Trashcan icon next to them. Usually Image Layers and Video Layers are will have this Trashcan icon. For Text and Outline Layers you can simply hide them clicking the Eyeball icon next to them.