It has been a really long time since I have looked under the hood of a gas powered loadstar so I will have to rely on a fuzzy memory.
Iirc, there is a very large vacuum line coming off the back of the intake manifold. That line will go towards the hydra-vac and the vacuum resevoir. You will also have the more normal vacuum lines that operate the governor (if so equipped) and the vacuum advance on the distributor.
If any of those lines had a leak it would show up in poor engine performance. The engine would not want to idle well at all unless the idle speed was really kicked up, in excess of 1000 rpm's .
If however the engine doesn't change in performance except for when you apply the brakes then you might be able to deduct the problem is in the vacuum diaphram in the hydra-vac unit.
But if the engine doesn't ever really change then I would point the finger at the vacuum reservoir.
Normally you should be able to hear a vacuum leak. But in my experience a vacuum leak, unless it was really large, would not be heard over the engine and road noise--loadstars are no where close to the "can even hear the clock tick" ambiance you will find in other makes.
I would not be too worried about a bad hyrda-vac unit. Most of them are pretty troublefree, even after fifty years.
From your description it sounds as if your hyrda-vac unit isn't getting any or enough vacuum to operate correctly.
Again, if your engine runs smoothly all the time I would suggest you most likely do not have a vacuum leak but a problem with your vacuum system going to the hydra-vac unit. If the reservoir is all gummed up it won't hold enough vacuum to assist the hydra-vac.
The reservoir can get all gummed up for a lot of reasons, most of them age related.
One way in which to eliminate the hydra-vac as the problem is to disconnect the vacuum line(s) to it. If nothing changes then you know the problem is upstream from the hydra-vac. If the engine won't idle you know you have a clear shot up to the intake manifold and the problem is in the hydrac-vac.
If you want, contact me directly and I will see if I can help.