Here’s something Grandma Jard used to say: “A weed is just a plant in the wrong place.”
So one, if something looks nice where it is, you can probably leave it there! If everything looks good, it probably isn’t competing with the plants you already put there, and as long as you keep order, everything will be fine. It doesn’t matter whether your neighbour calls it a weed – if you like it, keep it. It clearly wants to grow there, which is more than you can say for a lot of other plants!
Secondly, if you look at something and it’s out of place – something you don’t recognise or something which spoils the pattern of your garden – then you know it’s a weed. No matter how rare it is or how difficult it might be to grow, you can feel totally justified in getting rid of it. (Though in some cases, you might want to try and move it to another part of the garden!)
We strongly believe that weeds are what you make of them so just go ahead and pull the ones you don’t like, and leave the ones you do!