Bernie Madoff 'Can Live With' Fraud Victims' Anger, But Not Family Scorn, He Tells Barbara Walters Exclusively

Oct. 27, 2011 — -- Convicted Ponzi schemer Bernard Madoff was forced to "let ... go" of his wife almost a year ago and is wracked by "horrible nightmares" as he sits in a North Carolina prison, he told ABC News' Barbara Walters in an exclusive interview.

Though he "can live with" the anger of people he defrauded out of billions of dollars and he is adjusting to the rhythms of life in prison, even at 73 years old, he is troubled by anger and turmoil within his own family.

"Not seeing my family and knowing they hate me" is the worst thing about being in prison, he said. "I betrayed them."

Asked what he'd like to say to his grandchildren, he said, without apparent emotion, "I am sorry to have caused them pain."

As he sat across from Walters during a two-hour conversation at the Federal Correction Complex at Butler, N.C., on Oct. 14, Madoff often seemed to be trying as much as possible to feel no pain, himself.

There were no pictures taken of the session because cameras are not allowed into the prison for interviews with Madoff.

Madoff said he passes the time by reading, recently finishing a book about Wall Street robber barons.

The man who ran a Ponzi scheme of more than $60 billion has held six or seven different jobs in prison, he said, and he makes $170 a month.

He said he is relieved to be free from years of fear he'd be discovered as a fraud and finally has overcome thoughts of suicide.

"I feel safer here than outside," Madoff said. "Days go by. I have people to talk to and no decisions to make. ... I know that I will die in prison. I lived the last 20 years of my life in fear. Now I have no fear -- nothing to think about because I'm no longer in control of my own life."

Repeatedly throughout the interview he told Walters that he was guilty of the crimes that put him in prison, saying "I deserved to be punished. I deserved to go to jail."

Bernard Madoff Estranged From Wife

Though Madoff has people to talk to in prison -- the young prisoners look up to him for all the wrong reasons, he said -- his family situation is far more complicated.

He has not spoken to his wife, Ruth Madoff, since after the suicide of their son, Mark Madoff, on Dec. 11, 2010. And Mark Madoff's widow, Stephanie Madoff Mack, has told ABC News she holds Bernie Madoff responsible for her husband's death and, "I'd spit in his face," if she ever saw him again.

Madoff told Walters that his wife used to visit him at the prison weekly and they spoke on the phone daily. In order to visit Butner, N.C., Ruth Madoff would drive 12 hours alone, stay at a motel overnight and drive 12 hours back to Florida, which was hard on her.

But after their son's suicide, the couple had an emotional final meeting at the prison at which Ruth Madoff "asked me to let her go, which I understood," Madoff said.

"Ruth not communicating is the hardest thing," he added. "Ruth doesn't hate me. She has no-one. It's not fair to her. She lost her first son. ... She is a devoted wife and didn't care about the money."

Madoff told Walters he has not reached out to his wife since that final meeting.

Bernard Madoff on Clients' Anger: 'I Can Live With That'

Madoff seems to have fewer worries about the victims of his massive fraud.

"I understand why clients hate me," he said. "The gravy train is over. I can live with that.

"The average person thinks I robbed widows and orphans," he added. "I made wealthy people wealthier."