The RPC runtime caches call-related information so subsequent calls are be faster than the first. This cached call information is keyed by the identity associated with the binding, so if you change the authority information on the binding, the cached call is not reused for the next call. If your application switches back and forth between a few identities, you can improve performance by keeping a separate binding for each identity, instead of resetting the same binding.
For the same reason you want to conserve the identities that you use. Delegation creates new identities. These identities are several KBs in size and are cached for about a minute. The rate at which your application creates identities determines how many, on average, are in the cache.