Well, I lost my first one (the iPlay one) on my very first ride (long ride, not around the block) and I suspect it was due to me not tightening the rear axle enough after removing the wheel to apply the paint protection tape.
The one I made has stayed put now for well over 400 miles. The advantage to the new one is you can actually use the set screw on the side of the dropout to hold the nut in place as well. If it did come loose for some reason, with the set screw there, it will not fall out onto the trail. See the pictures in the original post. The iPlay nut was too small to make this set screw functional.