10-32 and M5 are almost identical. If you're not putting much weight on the rails, you can force one into the other. It will result in minor damage to both the bolt and nut, but it works.
To measure the diameter of screws and bolts, you measure the distance from the outer thread on one side to the outer thread on the other side. This is called the major diameter and will usually be the proper size of the bolt.