Right click on the cropping rectangle and hit Restore. If you have set any keyframed animation, delete the keys you don't want by right clicking them and selecting delete.
It is keyframe specific, to remove a pan/crop, just make sure you highlight and select the keyframe at which point that you want to remove pan/crop, right-click and select restore as suggested by cheesehole or simply select {default} from the drop-down list.