Jesse James said that the infamous photo of him giving a Nazi salute was just a "joke," but he no longer finds it funny now that his marriage and reputation has been "decimated," and he's lost the infant son he adopted with Sandra just months before the cheating scandal broke.

"I could tell by the look on my face it was a joke that was funny then, probably for a minute, but then looking at it in the context of now and in my life, it's not funny... " James said during a clip from the Nightline interview that airs tonight. "There's not a racist bone in my body."

"Dealing with losing my marriage and my son and embarrassing everyone and decimating my life..." Jesse said, sadly, "but to be called a racist on top of that is, it makes me really sad."

And when asked why he and Bullock adopted a black baby, Jesse said he "didn't care what color" the baby was. "My only prerequisite for adopting a baby: I want the baby that needs us the most."

James also said that when he told Sandra he was going to rehab, she told him she was "proud" of him and then promised to look out for Sunny, the daughter he had with ex-wife Janine Lindemulder.

But since even more affairs have been revealed, Sandra hasn't been seen with Sunny, who's living with her dad in Huntington Beach, while Bullock takes care of baby Louis in New Orleans.

Jesse has said he's "100% sure" Sandra will divorce him, but will she allow him to remain in her son's life?