The 56 year-old man was arrested shortly before the former White House strategist appeared as a guest speaker at the News Xchange conference.
Fellow protestors claimed on social media that he was held for holding the sign featuring a crossed-out swastika.
“It seems that actual Nazis are allowed to speak on our states but we can’t tell them they’re not welcome.”
A small disturbance broke out as the man was put in the back of a police van while shouting that he had been arrested for breach of the peace.
Police Scotland did not give further details about the alleged offence of threatening and abusive behaviour in its statement about the incident.
“A 56-year-old man has been arrested and charged with threatening and abusive behaviour at a protest in the centre of Edinburgh,” said a spokesperson.
“The incident happened around 2.40pm on Wednesday 14 November in the Morrison Street area. A report will be been sent to the Procurator Fiscal.”
Mr Bannon, a key figure in Donald Trump’s election campaign, has championed far-right nationalist parties across Europe since leaving the White House in August 2017.
Around 100 people gathered outside the Edinburgh International Conference Centre before Mr Bannon was interviewed on stage by BBC journalist Sarah Smith
Organisers claimed that it was their “journalistic responsibility to share and scrutinise a range of relevant viewpoints within the framework of a balanced debate”.
Further protests are expected when Mr Bannon speaks at a meeting of the Oxford Union debating society at Oxford University on 16 November.