"A scandal with that woman would mean ruin for me."