several things can cause that issue.
I don't remember "which" distributor you have. So let me know. For help in identifying various distributors used on IH apps, see this thread:
http://www.forums.IHPartsAmerica.com/ignition-tech/642-ihc-vehicle-distributor-identification.html
Trev is correct about the distributor endplay...but that doesn't show up when the rig is just at an idle...that allows the timing to "walk" continuously depending upon if engine is being accelerated or decelerated, has to do with "thrust" of the shaft from the helix of the drive interface. This can lead to random "detonation" due to unstable timing, that in turn can create some real issues mechanically with any engine that is running like that over time.
When running at idle, and the timing is "walking" as can be seen with the timing light, that is normally caused by a badly worn, but most likely "sticking" mechanical advance mechanism, the components that control the position of the breaker point cam. Again...the "fix" for that is a simple distributor overhaul or swap for a reman.
Also...the breaker plate mounting itself can be loose, or the "bearings" between the upper and lower breaker plate are missing (at least one of 'em).
Same for the vacuum advance unit, that needs to be tested for both proper actuation and if it will actually retain vacuum. See this thread for a test set-up, pay attention to post #5:
http://www.forums.IHPartsAmerica.com/tool-talk/1570-mityvac-thread.html
A vacuum advance unit that won't actuate and hold vacuum, is in reality a vacuum "leak" and affects drivability in many ways.
I service all types of distributors weekly (I'm doing a Ford/Holley today), but I don't have the ability to renew bushings, that work has to be farmed out to a rebuilder. However...normally the problems you describe are not caused by bushing/shaft "wear". They are caused by neglect over time. Refreshing the parts of the distributor above the main body is not a big deal, however...each version has it's own method of retaining the advance unit to the shaft...in the case of a Holley, removing the retainer clip without destroying it is a real bitch. And that clip is not a part that is readily available.
Also...a simple "check"...the rotor itself could be wrong for your distributor and be slopping around on the hub. Very common, and even the center conductor on the rotor can be "loose" depending upon which rotor and manufacturer's part it is. Same for an improper cap application, see this thread:
http://www.forums.IHPartsAmerica.com/ignition-tech/1567-prestolite-Holley-parts.html
Can you post some pics of your distributor/rotor/cap combo so we can tell more about it?