"An innocent man would've punched me by now."