you can get a small bearing and place it between two rubber washers. the first washer will cause it to grip the wood. the second will cause it to grip the metal arm. this will cause smooth action in your arm. you can also do this with soft polyurethane washers. did the same thing on a desk that i constructed years ago in wood shop class.
just be sure the bearing splits down the middle, looking rather like a sandwich, and that it isnt a ring rolling around the center, like a wheel.
alternately, you can get a washer that has fans or teeth on one side. these will imbed in the wood, and the metal arm will slide easily on the flat side of the washer. go this route if you can not adjust the arm to accept the bearings, or if you can not find them. the thing you are looking for looks similar to a dry wall molly bolt, if you know what those are... but only the head of it, not the whole shaft assembly.