The fact is, we know that pesticides, herbicides, and fertilizers (even natural ones) are poisonous to the environment. We know that fertilizers cause algal blooms which decimate aquatic fauna. We know that arsenic is a deadly environmental toxin. We know that herbicides have unintended side effects when they spread beyond the neatly-defined borders of farmland.
What we don't know is what the genes of genetically modified organisms do once they're out there. Bacteria readily take up "excess" genetic material and happily swap plasmids with each other, but that doesn't mean they have to actually use the genes. Genes, after all, take energy to maintain, so if there's no need for a bacteria to possess a gene against rice fungi, it'll probably discard it eventually. Even if the gene does persist in the bacterial pool, the mere presence of it is not enough to do anything; a gene that remains quiescent for all eternity may as well not be there, as far as we're concerned. To put it another way: cars are noisy, dirty, pollutants of city streets. But if it just sits there in your driveway, it's not being noisy, dirty, or polluting (I'm not going to go into the idiocy surrounding the "carbon footprint" question). In fact, from the environment's point of view, a car that just sits in your driveway may as well not exist.
We also don't know how these rogue bits of genetic material will affect the surrounding wildlife--indeed, whether or not they will have any effect at all. They could render some plants inedible to wildlife, or increase their abundance to the point where there's a population explosion of herbivores. They could enable a plant to spread far beyond its native territory, and become an invasive species elsewhere, or they could drive it to extinction.
But we just don't know. The Law of Unintended Consequences has made quite clear that overlooking the smallest detail can lead to the biggest f*ck-ups; the corollary to that law is that the detail that's overlooked is the one that nobody knows about until it's too late.
On the other hand, what we do know is that using chemical fertilizers, poisonous pesticides, and herbicides is a) bad for us, b) bad for wildlife, and c) uses unsustainable technologies and produces tons upon tons of chemical waste. You can't have your cake and eat it too: either have farmers use conventional methods to farm but poison themselves and the world, or increase their capacity for going organic by allowing the unbridled sale of GMOs.
It is my belief that life can handle a few new genes being tossed around. Life is, after all, 3.5 billion years old. It's handled mass extinctions, to levels we can't even begin to fathom, the build-up and breakdown of millions of species, over millions of years. We're not going to inadvertently cause the next mass extinction by the accidental release of a few plasmids.
As Sherlock said to Watson, "It is a capital mistake to theorize before one has data." But if one has data and chooses not to act on it, isn't that even worse?