Using average player height is nice and all, but it overlooks the fact that we don't really care so much about our guards getting taken advantage of by the average NBA player. We care about how they fare against the most dangerous offensive players. Those guys tend to be 6'3'', 6'4''.
I think Smart and Bradley are plenty big enough to handle most point guards, regardless, but do give up some height covering bigger shooting guards (think Klay Thompson, Kyle Korver, Wes Matthews, Demar Derozan etc), which makes a difference, though not a gigantic one.