Tom, both the "Register of the Commissioned and Warrant Officers of the Navy of the Confederate States, to January 1, 1863" as well as the "Register of Officers of the Confederate States Navy, 1861 - 1865" show his state of birth as Vermont, state of appointment as North Carolina, and year of appointment in the USN as 1857. On the assumption that middys were usually aged about 14 to 16 when appointed to the Naval Academy, I took his birth year as about 1843, which, together with the fact that the "James E. Fisk" shown in the 1880 census was born in Vermont, and no other Fiske or Fisk was born in the same state, and of the correct age, I took this person as being the same one.
Unfortunately, I cannot find a listing for a Vermont born James E. Fiske in either the 1850 or 1860 census, so I am unable to corroborate other facts.