Check event log for errors - it might help you to figure out why it doesn't work as expected.
You could remove web part in a following way:
Go to Page Templates app -> navigate your template -> Web Parts -> read the xml and find web part you want to remove -> remove the piece of xml responsible for the web part.
NOTE: This dialog should only be used for emergency reasons. If you want to modify the web part settings, use the Design mode of the page.