Some further info on James Edgar Fisk:

1850 Census - VT - Franklin County - Swanton [p 257 or 513 or (Ancestry.com's designation 17 of 68)]
James E. age 9
Zoraster Fisk age 46 Merchant, Real estate 1500
Ann M. age 37
A Margret Riga age 24 (Canada) is also listed for the same household, perhaps a servant.

1860 Census - MD - Anne Arundel County - Annapolis p 95
Listed on line no. 18 as J. E. Fish, age 18 Midshipman Vermont. He is right between J(ohn) McFarland and W(illiam) B(arker) Cushing.