Caffeine is a vasoconstrictor so it can make your flaccid measurement appear smaller. In high enough dosages or in men with ED it may interfere with erections.
To expand on this, the effects are temporary. The half-life in most adults is about 5 hours.