I was assuming that during the Twins scene when Strucker was complaining about the scrutiny that Clint was already in the process of breaking in.

I think the fact that Hydra had Loki's scepter would piss him off alone but prisoners just make it worse.