How Do We Separate Historic From Fantastic Language in Genesis 1-11?

Well, I don't know that there are any really hard and fast criteria, but I can think of a couple that could be applied. One would be when the narrative affirms something that we know that the Pentateuchal author did not believe. So, for example, when in Genesis 2 and 3 you have God described in these humanoid terms – making a statue out of dirt and blowing into its nose, or doing surgery on Adam's side, or walking in the cool of the afternoon looking for Adam hiding in the bushes – these are incompatible with the portrayal of God in Genesis 1:1 who is the transcendent creator of the whole universe and not some sort of a physical, much less finite, object. And so that would be I think a dramatic tip off to us that the author intends these narratives to be taken figuratively. Another good example would be the days in Genesis 1. I don't think that the Pentateuchal author would have believed that the primordial ocean covering the Earth could have drained away in just 24 hours. For that to happen we'd have to imagine the water miraculously speeded up and rushing across the continents and into oceans and lakes and things. That'd be a catastrophe, and that's surely not what the author is envisioning in his mind. Look at the flood story in Genesis 7. The flood of Noah returns the Earth to its primordial state of being completely submerged beneath this ocean, and how long does it take for the waters to drain away? He says 150 days before the mountaintops were visible, and even then Noah isn't allowed to exit the ark; it takes more time for the water to drain away. So this would be a great example, I think, where the Pentateuchal author himself wouldn't have believed that this was a literal 24-hour period of time and that therefore this is likely to be metaphorical. And similarly with others of the days in the creation account. So that would just be one of the signs that something is meant to be figurative rather than literal – because it would contradict what the Pentateuchal author literally believed.