I don't think Tofu plays a role.
Perhaps it is wise to limit processed soy foods, like vegie burgers, to perhaps once a week or so. Soy foods the traditional way though, like tofu and especially tempeh are safe to eat and enjoy in moderation. You probably should not eat an entire block yourself everyday as some fitness gurus advise. The Okinawans, the world's longest living population, are also the world's largest consumers of soy (do you suppose there is a correlation?)but even they don't eat that much.
There is evidence for and against soy, but there is far more evidence on the side that says soy is a very positive addition to a balanced diet. Soy, the traditional way, and especially fermented soy foods are healthful and delicious.
As to phytoestrogens causing you to have a small dick....no I think you just have a small dick. So, leave soy alone.
I do know what you are talking about though. Phytoestrogens are real - but they don't hurt you for the most part. After eating a block of tofu after a workout, some men report their faces looking softend, and less lined, almost "pretty", with a sparkling complexion they next day. This is generally considered a positive side effect, but it shows the effect of phyto estrogens. Soy helps your skin through a number of other processes, but one of the ways is the exact opposite of what you see when athletes naturally or chemically attempt to raise testosterone and male hormones. It is the "hard" look you see on their faces, clasically on the faces of women bodybuilders on steroids. If they ate soy - they might soft and feminine, like the asian beauties we all admire so much. Remember though, that this is a very mild and temporary effect. The tofu skin effect never makes a man look feminine, only polished, and this only happens in some individuals. Furthermore, for it to even happen, you need to eat an aweful lot - much more than you would in a typical meal, and then all at once. It can never actually make a man feminine. Soy's phytoestrogen action is very moderate, and not harmful, even in large quantities to the vast majority of people.
There has been some research that has shown that excessive soy consumption can lower memory testing scores (directly after consumption) in older Japanese men, and there is some theorizing that it may have to do with the effect on sex hormones. Again though, moderation is key.
Now, to the important point. Soy, and other plant foods, do not contain estrogen.
That's right, they do not contain estrogen. Phytoestrogens as natural molecules produced in plants. They mimic estrogen. They fit into the estrogen receptors in animal cells, the same way a smaller key fits into a larger lock, usually opened by a larger key. Phyto estrogens are the small key, estrogens are the larger, real key. They do not do the same thing as estrogen exactly. However, they keep real estrogen from locking into those receptors. So... they may be protecting men from much more dangerous kinds of estrogens/oestrogens.
Do you know what does contain estrogen and other hormones?
Beef does. Milk does too. You know, the Mama Cow? So before you fly off the handle, renounce healthful foods, and start out on a diet off the not-so-fresco menu at taco bell, put things into perspective. Remember, I first of all recommended that nobody go gung ho on processed soy, or soy foods in general. I would much rather though, that you ate a boca burger everyday, than a cow burger everyweek. It is safer. Leaving aside all the planetary and ethical reasons, animal foods simply contain much much more in the way of hormones, chemicals, pesticides and synthetic and biological contaminents.
Meat for example, typically has 9 or 10 times the amount of pesticide residue as plant foods. Dairy foods rank in at about 4 or 5 times the pesticide of plant foods.
Soy - whatever it's special properties (and all plants have them, healing and otherwise) - is at the bottom of the food chain. Eat from anything above that, and you are getting a progressively concentrated dose of whatever truly dangerous environmental ostrogens are out there.
You can look for a reason to explain why you are gay, or your penis is not as big as you think it should have been, or why you are having any other problem in your life - and if that helps you, go for it. Ultimately though, we all have stop overanlyzing and take action to make the best of whatever situation we find ourselves in and thrive in that place.