"For two bits I’d shove out of here"

A ‘bit’ refers to a unit of money, in the USA to 1/8th of a dollar. ‘Two bits’ is therefore a quarter of a dollar.

There is no actual coin or paper money that represents a bit, but the expression has remained in usage from the days when there was.