Up to what point does one have to wash hands when touching his arms?

See question 2997 regarding washing hands when putting on Tefilin she Yad. We mentioned that Horav Shlomo Miller’s Shlit’a maintains that he usually washes his hand after putting on the Tefilin. However, when it is just bellow the elbow, no Netilas Yodaim is needed. The elbow itself is debatable. However it would seem that since the common summer shirts leave it exposed, one does not need to wash hands when touching it.