"the ancient palace of King Nicomedus, where voluptuous memories of the young Julius Caesar abound"

The reference is to King Nicomedes IV (reigned 94-74 BC) of Bithynia (north-west Turkey). As a young man, Julius Caesar had been sent as an ambassador to Bithynia to collect funds to raise a fleet. According to Suetonius, he remained there for so long that rumours began to circulate of a homosexual relationship between him and Nicomedes, rumours that would be used against Caesar by his enemies for the rest of his life. Hadrian goes on to discuss his own homosexual relationship with Antinous, which begins in the same palace.