throw the baby out with the bathwater
Short definition from Wiktionary
Verb
- To discard, inadvertently or through overzealousness, something valuable or essential ("the baby") in the process of removing or rejecting something unwanted or undesirable ("the bathwater").
We haven’t written up throw the baby out with the bathwater yet: Simple and Standard meanings, examples and a quiz.