The president and first lady emerged from the White House shortly before 10 p.m. to watch a fireworks display that ended the inaugural festivities on Wednesday night.

While the Bidens were on the Blue Room Balcony, the vice president and second gentleman watched from the steps of the Lincoln Memorial, where Harris had spoken to the public a few minutes earlier and where Katy Perry was performing her song, "Firework."

In a show of unity and bipartisanship, the three former president who attended Wednesday's inauguration recorded a video while they were at Arlington National Cemetery earlier in the day.

Former President Barack Obama kicked off the discussion, noting his pride in watching his former vice president and the first Black, South Asian American and female vice president being sworn into office. Then the group turned to address the day’s theme of unity.

“I think that the fact that the three of us are standing here talking about a peaceful transfer of power speaks to the institutional integrity of our country,” said former President George W. Bush.

He also called on Americans to love one another the way they'd like to be loved, while former President Bill Clinton said more directly, that "everybody needs to get off their high horse and reach out to their friend and neighbors to try to make it possible."

The three former presidents ended the video by offering some parting words to Biden.

"Mr. President, I'm pulling for your success. Your success is our country's success," Bush said, paraphrasing a message his father left for Clinton during their transition.

"I'm glad you're there," Clinton said. "And I wish you well. You have spoken for us today. Now you will lead for us and we're ready to march with you."

And Obama told his former vice president that they’d all be there for him.

“Joe, I'm proud of you. And you and Kamala need to know that you've got all of us here rooting for your success, keeping you in our prayers, and we will be available in any ways that we can as citizens to help you guide our country forward," Obama said. "We wish you godspeed.”

-ABC News' Molly Nagle and Justin Gomez

Vice President Kamala Harris addressed the public for the first time since taking the oath of office during the "Celebrating America" inaugural program Wednesday night and she echoed Biden's call for unity and overcoming adversity.

"In many ways, this moment embodies our character as a nation. It demonstrates who we are," Harris said. "Even in dark times, we not only dream, we do. We not only see what has been, we see what can be. We shoot for the moon, and then we plant our flag on it. We are bold, fearless, and ambitious. We are undaunted in our belief that we shall overcome, that we will rise up. This is American aspiration."

The vice president referenced Abraham Lincoln's work with land grant colleges and the transcontinental railroad during the Civil War and Martin Luther King Jr.'s fight for racial justice during the Civil Rights Movement as aspirational moments.

"American aspiration is what drove the women of this nation throughout history to demand equal rights and the authors of the Bill of Rights to claim freedoms that had rarely been written down before," she said. "A great experiment takes great determination. The will to do the work and then the wisdom to keep refining, keep tinkering, keep perfecting." 

This "same determination," she continued, "is being realized in America today," through the work of scientists and parents nurturing the next generation of educators and innovators. 

"This, too, is American aspiration," she said. "This is what president Joe Biden has called upon us to summon now. The courage to see beyond crisis, to do what is hard, to do what is good, to unite, to believe in ourselves, believe in our country, believe in what we can do together."

Former presidents Bill Clinton, George W. Bush and Barack Obama highlighted the importance of a peaceful transfer of power in a new video message set to air Wednesday night, ABC News confirmed.

The video, organized by the inaugural committee, features a conversation with the leaders about upholding democracy and staying civically engaged.

The three were the only former presidents at the inauguration ceremony. Jimmy Carter was unable to attend and Donald Trump left the White House ahead of the swearing-in.

-ABC News' Cecilia Vega and Mae Joo

Biden felt like he was "coming home" after being sworn in as president Wednesday, according to the White House press secretary.

"Remember, he spent eight years here as the vice president, playing an important role as a partner to President Obama. And, you know, that was the emotion that overtook him today," Jen Psaki said during a press briefing when asked what it was like for Biden to be in the Oval Office.

The president also felt an "incredible sense of calm" and "joy" on Wednesday, she said.

"He spent the day with his family and his grandchildren and his children, and that always has an impact, I think," Psaki said.