I cannot answer in general, I'll just describe a bit how its handled in EnsoMOSAIC.
Many times exterior orientation is not accurate, especially for small format and unmanned systems. This initial inaccuracy is taken into account by setting initial weights of exterior orientation relatively low compared to those of tie points or ground control. Realistic weights can then be determined by an iterative process, evaluating post-adjustment residuals for setting pre-adjustment weights.
Tie points locations are not determined in block adjustment but before it. During the adjustment points with high residuals are automatically eliminated from the result. This is carried out by dividing the block into sub-units for uniformly accurate process.