The men who established the republic were no plaster saints of Red State moral uplift. Only one of the half-dozen figures Thomas A. Foster writes about in Sex and the Founding Fathers: The American Quest for a Relatable Past (Temple University Press) would escape denunciation by the Traditional Values Coalition if the Founders were around today.
Accusations of adultery or of fathering children out of wedlock (or both) were made against George Washington, Thomas Jefferson, Benjamin Franklin, and Alexander Hamilton; the last two admitted the truth of the charges. Gouverneur Morris managed to draft the Constitution between rounds of frequent, strenuous fornication -- exercise he pursued despite having a severely mangled right arm and amputated left leg.
Only the the tightly wound John Adams seems to have escaped any hint of scandal. By all evidence, he and Abigail were strictly monogamous and not averse to finger-wagging at the other Founders' morals -- especially Franklin's, which were particularly relaxed. Besides writing a notorious essay on selecting a mistress, Franklin lived with a common-law wife; later, he conducted a good deal of his work as ambassador to France either in bed with well-born Parisian ladies or trying to get them there.