Loose clothing has a tendency to get in the way when you least want it to and it's more likely to catch on branches and thorns. Some loose hiking shorts, depending on what they're made from, can also swish as you walk which may or may not annoy you, but it does make it more challenging to sneak a peak at wildlife without startling it.
Waistband fit is also important, especially if you plan on carrying a backpack. Waistbands that are loose and fold over or bunch-up will only serve to irritate you. Also, if you're backpacking you may want to consider hiking shorts with a belt, as they won't be able to move around as easily. Hiking shorts that "travel" or that are loose enough at the hips/waist that they need to be pulled up every few minutes are not only annoying, they're also more likely to chafe and might allow those following behind you to see a side of you they may not have seen before. ;-)