1963 Nyer Brake issue

Question for the all experts here.. I have a 1963 Nyer 4 door.. with single master cylinder. Brakes work fine. However, when I depress the brake pedal.. the idle noticably drops/pulses/becomes irratic. Once I release the brake pedal... it returns to normal. Any suggestions? Thanks in advance.
Author: blue88