Um, consider the average battery. Now where are you going to apply the field, in which direction, and how will it cause current to flow?
First off, voltage is just the path integral of the electric field. If the top terminal and bottom terminal of a battery are at different voltages, then there's an electric field between them. Casing and an internal electrode bend this electric field so much of it ends up radial, but it's there, and it's still between the top and bottom terminals. If you apply an electric field along this direction, then the voltage differential between the two electrodes will change, and the internal electric field will change too. Depending on which way you apply the field, the response will change.
Inside the battery, you have a chemical reaction that pushes charge in one direction, driving the voltage difference. This chemical reaction can only procede when the voltage drops below a certain amount, and may proceed in reverse if the field gets much above this value (the recharging process). There is also a small leakage current within the battery, as charges get pushed back because of the electric field it creates. That's the slow self-discharge. Current flows, decreasing the voltage, and the chemical reaction can then proceed until the voltage is re-established. It's an equilibrium process.
If you apply the field in the same direction as the battery's existing field, then this higher electric field will create a higher leakage current. But this extra leakage current will also build up a screening charge in the process, and so you only get a little bit of extra leakage current before the field is screened, and then nothing more. You will also get a bit of the reverse of the chemical process, which will recharge the battery slightly. When you remove the field, the voltage will be too low, and the chemical process will go forward again until it re-establishes the voltage difference. These will all be minor changes, though.
If you apply the field in the opposite direction, then the voltage difference between terminals is lowered and leakage current is discouraged. But the chemical reaction faces a lowered voltage and so will proceed until it, too, builds up a screening charge that re-establishes the voltage difference between the two sides. When you remove the external field, the voltage will be too high, and you'll again get a combination of leakage current and reverse chemical process.
In both cases, the net charge/discharge from the whole process might be zero (compared to the normal leakage current that's happening continually). But instantaneously, you can still be driving currents around, which really shouldn't surprise you.