Claudius is talking about Calpurnia, the prostitute with whom he was living. In the Author's Note at the start of Claudius the God, Graves asserts that "no character is invented in either volume." He is, at the very least, stretching the point. A person named Calpurnia is, indeed, mentioned by Tacitus as one of Claudius's mistresses, but nothing that he says would allow this profile of her to be built. This is pure (and glorious) fiction.
Claudius's statement that he took Calpurnia into his confidence and never regretted it later becomes significant, although its full relevance is only clear when the two books (I, Claudius and Claudius the God) are considered together. Here Calpurnia is shown as politically and financially astute, predicting the disaster that Caligula's excesses will visit upon Rome long before anyone else does. Her preference for cash rather than Claudius's offer of a pearl necklace will later take on an element of poignancy.
Philo visited Caligula in Rome as part of a deputation of Jews, protesting the Emperor's insistence that he be worshipped as a god (a bigger problem for the monotheistic Jews than it was for adherents of most polytheistic religions around the Empire). In Graves's fictionalised account, Caligula wakes from his long illness convinced of his own divinity.
Graves is having fun with the interpretation of prophecies. Livia, on hearing Thrasyllus's prophecy about the emergence of an everlasting God, believed that it referred to herself. Caligula now convinces himself that it relates to him.
Cassius Chaerea is mentioned by Tacitus and Suetonius, so he is not an "invented" character, but his persona in the novel is largely invented by Graves. Tacitus tells us that he distinguished himself in the German campaign, but Graves invents the story about him being a survivor of the Teutoberg Forest massacre, along with his role in the mutiny at Bonn, his early encounter with Caligula and his feat of bravery in the amphitheatre. Like Calpurnia, he becomes a significant character later in the story.
Claudius admits that he has no proof for his contention that Caligula murdered her, and nothing in the ancient sources suggests that he did so. However Suetonius does make the allegation that he had previously raped her, and had had incestuous relations with his other sisters also.
It is unclear whether Graves, when writing I, Claudius, was also aware of the archaeological discovery in 1927 of two enormous ships from Caligula's time on Lake Nemi. The largest of these appears to have been a floating palace, with baths and mosaic floors. The ships were being excavated and consolidated as Graves wrote.
Graves's account of Caligula's construction of a bridge across the Bay of Baiae is based partly on Suetonius's The Twelve Caesars and partly on Cassius Dio's Roman History. Suetonius's suggestion that Caligula thought this idea up "to outdo Xerxes, who excited much admiration when he threw a bridge over the rather more narrow Hellespont" seems like a deliberate spoiler, given the fact that Xerxes, far from being admired for his action, became one of the stock-examples of hubris (for example, in Aeschylus's The Persians).
Graves's Claudius suggests that Caligula's real motivation was rather Thrassylus's prophecy (that Caligula was no more likely to rule the Empire than he was to cross the Bay of Baiae on a horse). Nonetheless he has him laughing at Xerxes, and berating the god Neptune as a coward, promising "to teach the old god an even sharper lesson" some time soon.