There are such a thing as environmental oestrogens. It doesn't seem that the problem is so much that they make men have smaller penises though.
They are thought to be possibly connected to a higher incedence of intersex births, gynocomastia, obesity, infertility and maybe even being gay (yes controversial). They don't come from women. They are found in the environment, and particulary water. They are come from plastics, and other chemicals, fertilizers, hormones used in raising farm animals, drugs, and - well yes, women excreting certain amounts of hormones from birth control pills in their urine, and a certain amount coming back into the city water. There is not much known about them. As of now, we know that they interfere with the ability of certain species of fish to breed. It seems that whatever effect they have on mammals and humans is most significant while they are in the womb.
But, no, hanging around girls, or having been around girls a lot growing up can't make you have a smaller penis.