Breast milk is produced for the purposes of feeding babies. It's the healthiest thing to feed a baby, as opposed to formula.
The purpose of veganism is to reduce suffering of animals, live a healthier lifestyle, and to help preserve the environment. Breastfeeding does not negatively impact any of these things.
On the other hand, humans are the only species that consumes the milk of a species other than our own. What if your dog drank giraffe milk? Wouldn't that be weird? Same goes for humans consuming cow's milk.