Is an affair with your husband's caretaker okay?

My husband had a stroke awhile ago. I hired a helpful caretaker to assist him, but now I'm falling in love with the caretaker! That's okay, right? My husband was a bit of a gambler and drinker previously, if that helps.