Princess Diana's brother, Charles Spencer, says he was "really upset" after Buckingham Palace issued a rare response to his book, calling the response condescending.