I agree, to a point.
If it turned out to be a live shell permanent removal is not the only option to make it safe. Looking a the photographs it shows that someone patched around the ball with mortar. It may have already been taken it out of the wall and the explosive removed. Maybe there is a record of this. If no record exist, I would recommend removal, examination and if needed, making it safe to the public. It then could be reinstalled in it's place in the wall.
If it is a live shell the chances of it "going off" in the future are very low but there would a precentage of probability. It would be better that the chances for detonation are zero percent than to have anyone hurt or killed by an old Yankee shell.
If it were to explode, and someone was hurt, the sponsors of the Church would be sued, I am a full supported of sponsors of Civil War history and would hate to see them ruined by litigation.
Better safe than sorry.