As long as you maintain a good degree of penile "fitness" through stamina work and/or sexual activity (the requirement to maintain mostly being the ability to achieve and hold rock hard erections) then your tissues should remain expanded and your gains maintained.