I don't usually write articles in response to things elsewhere on the internet, but I felt a need to defend lolicon lovers, who have been getting increasingly stomped on of late. Links to what I'm replying to in the first paragraph. You should probably go read that page first, as well as the original post that sparked the debate. I'll be right here when you get back.
Here, in the first submission/response, the submitter does, in fact, say in the same breath that it's not real and doesn't invoke the same feelings and argues against age of consent laws. I understand why AnswerMan feels that argument is disingenuous. And yet I disagree with him.
It's a common thing. The same thing happens, for example with the "downloading is stealing" debate. In the same breath I would say that downloading is not stealing because it doesn't result in a physical loss, but only a perceived loss in terms of intellectual property, and that I disagree with, not necessarily the spirit, but the way copyright and DRM are being used to shepherd people around within a given vendor's hardware-and-software-and-content rolled into a single product. I disagree with the means of distribution, so I obtain the content in a DRM-free format. In one manner it's a form of rebellion, and in another, a means of getting my cake and eating it too.
I think the "fantasy" of sex with children very much does extend to real children. The fantasy, mind, not the reality. People fantasize about dinking their peers all the time, be it some hot chick in school, or at work or wherever. I don't think that's any different for these people. But reality never enters that picture. If given the opportunity, I'd like to think I wouldn't carry out most of my fantasies, regardless of whom they involved. The fact that there are people out there exploiting real children is despicable, even to lovers of this medium. I say this generally--I'm sure there are also exceptions. I don't, however, think it peculiar that they at the same time mention age of consent laws. I read his rant. He wasn't arguing that age of consent laws should be changed. In fact, he was arguing that society's perception of what is and isn't "right" as regards sex and age is skewed. There isn't a one-size-fits-all rule. It would be like saying a person must be at least this tall in order to consent to sex. Height is, after all, another variable generally directly proportional to maturity, is it not? As one increases, the other (generally) does as well. But it obviously isn't a clear indicator of how mature a person is. You can't quantify one from the other--and so it is with age. Your assertion that he would think middle aged men should be able to have their way with children based on that is flawed. No such argument was even implied. We don't know how "mature" a person is, sexually, based on their age. That is the problem. No solution was offered by him, and I have none to offer either. It was simply a statement of fact: society doesn't know how to best protect the innocence of its youth (look at debates on sex and violence in movies and videogames--everyone's just talking out their ass on both sides), and so it tries to protect by default (whether the protected want the protection or not) and any deviants, even in thought, are labelled perverts. Well, as often happens in history, what's slung as an insult is often taken as a compliment and even worn as a badge of honor. Deviants are generally proud to be such.
Much like other "alternate lifestyles" (lovers of BDSM and other sexual fetishes that involve bodily fluids most people are too polite to mention, spring to mind), they just want to be left alone. They don't really want you to understand them--I think for the average person, their mind doesn't work that way and never will--they just don't want to be looked upon strangely for their preferences. It could be reasonably well argued that they shouldn't make it so public then--if you don't want people to take notice of you, you shouldn't be proclaiming what you are from rooftops. But that doesn't work if your aim is being treated equally. It almost sounds like I'm pushing a gay agenda here, but they don't want to be afraid to mention their preferences in casual conversation, to be always "in the closet" as it were, about the things they like, which seem to always be inexplicably inappropriate to mention, even amongst friends. As has been previously mentioned--if they're not hurting anyone, why the fuss? There aren't gangs wearing loli tshirts prowling for children in the ghettos are there?
Of course, maybe you see this as one more stepping stone in the erosion of traditional American family values (which everyone knows are the same for all Americans, *cough*) and degradation of our great society (if not American, substitute country of residence). Maybe you think to yourself "what next, Romanesque pederasty being socially acceptable?" The great civilizations of the past have been notably more lax about this sort of thing (glabraria being Latin for "lover of smooth-skinned boys" and Glabrarius being a common name in Rome--a bit of linguistic proof, nevermind all the historical evidence; in fact, this aspect of their society was noted casually in my high school Latin textbook). We don't want that in our culture--fine--but I see no evidence of that happening. Because such "evidence" usually takes the form of the following non sequitur (from the same page as above, last submission):
"I beg everyone here to recall the kidnapping, rape and murder of Danielle van Dam in 2002. We very nearly dodged a bullet as anime fans, inflicted by this lolicon garbage. David Westerfield was found to have had lolicon on his computer. I hope I need not tell you his connection to the case." Yes, and the Columbine kids were discovered to have played a lot of Doom on their computers. "Oh noes! I see causality where only correlation really exists!" Please. Also from the same posting: your principles are not jeopardized by their desired privileges. You have exactly the same course of action available to you that they are exercising. If you don't want the stigma of being associated with them, then do what they're doing: get on your rooftop and start declaring your love for anime that does not include lolicon. Make it clear to the world that the two are not synonymous. It is your obligation to make the distinction, not theirs. Just be careful not to slander the other side. There's a world of difference between "I love lolicon! Leave me alone!" and "I hate lolicon! Get rid of it!"
Lolicon lovers aren't hurting anyone. You may not like the same things they like (as is the case with most niche markets)--you may even despise it and/or them--but there is demand, and supply will follow. It is their right to put their money where their mouth is, in our capitalistic society. Unless, at some point, their behavior is deemed of social harm and made illegal, they can exercise that right. To the people who "wouldn't cry" if lolicon were banned, I'm quoting this especially at you:
"First they came for the Jews
and I did not speak out
because I was not a Jew.
Then they came for the Communists
and I did not speak out
because I was not a Communist.
Then they came for the trade unionists
and I did not speak out
because I was not a trade unionist.
Then they came for me
and there was no one left
to speak out for me."
and as a corollary:
"Most human beings have an almost infinite capacity for taking things for granted." --Aldous Huxley
And a few more relevant quotes:
"Fear of serious injury cannot alone justify suppression of free speech and assembly. Men feared witches and burned women. It is the function of speech to free men from the bondage of irrational fears."
--Louis D. Brandeis, US Supreme Court Justice
"I would rather be exposed to the inconveniences attending too much liberty than those attending too small a degree of it."
"A civilized society is one which tolerates eccentricity to the point of doubtful sanity."
It seems to me both camps are trying to maintain the status quo and to keep something alive they care about. Maybe they could work together, in the spirit of this sentiment:
"Monsieur l'abbé, I detest what you write, but I would give my life to make it possible for you to continue to write."
Voltaire, letter to M. le Riche, February 6, 1770
Keep the discourse alive. Remember: "The more we disagree, the more chance there is that at least one of us is right."
 As an aside: I would also argue, at this point, that it is in fact largely society's fault that child-love is as widespread as it is. Is it really a wonder that people are to be found who objectify minors, when media and entertainment (including, but not just, pornography) are constantly pushing younger models as more attractive models? A major sector of porn focuses on "barely 18"--i.e., as young as they can legally get away with. "Young" and "sexy" are almost interchangeable in our society. If younger == better, would it not logically follow that people are going to start exploring even underage content? And lolicon is a safe haven for such exploration, where one can do so without guilt, because its depictions aren't real and are unaccompanied by the physical and mental trauma associated with real child porn. I'm not saying this is good or bad--just that we brought it on ourselves. "A single person is everyone's fault and everyone is a single person's fault."
 "The legitimate powers of government extend to such acts as are only injurious to others. But it does me no injury for my neighbor to say there are twenty gods, or no God. It neither picks my pocket nor breaks my leg."
 Also as an aside: I would point out here that people are stupid. Nothing you do is ever going to make you immune to stupidity. People have a tendency to immediately and angrily "point the finger" externally. They don't want to admit that the shortcoming is theirs, and they want to paint with the broad brush of generalities and stereotypes. If they want to blame something on anime, or cartoons, or Japan, or things that move, they're going to.
"People demand freedom of speech to make up for the freedom of thought which they avoid."
 I see your Dwight Eisenhower:
"A people that values its privileges above its principles soon loses both."
and I raise you one Benjamin Franklin:
"Those who are willing to sacrifice their basic liberties to assure their security deserve neither."
On one side you have those who are expressing (potentially genuine) concern over the well-being of children. Both sides notably have the security of their hobby and passion they feel to be at stake (right or wrong). But only one side (the general-anime side) is talking casually about banning the other side's entertainment.
 That was a joke. Mostly.