David,

You asked, "the Federals withdrew from Baton Rouge???"

Yes. On August 21, 1862 the Federals evacuated Baton Rouge and the 4th Louisiana reoccupied it on the 23rd. On Dec. 17, 1862 the Federals moved up river and retook Baton Rouge without a fight. During the intervening time Port Hudson had been fortified, so there was little reason to fight for the city.

Van Dorn was a notoriously poor judge of military strategy. He was all "huzzah" romanticism and lacked a pragmatic approach. He tried to conduct grand attacks that called for his forces to cover too much ground and required too much coordination to succeed. His boldness only really served him as a cavalry raider; as an army general he was a failure.