The couple took advantage of the time when their children were away from home