Facebook's failing moderation team permitting their porn bot prompts to describe children is obviously bad, but I can't fathom why Facebook would want to be in the porn bot business ("romantic companion" my ass) in the first place. What's next, a MindGeek merger?
In turn, I don't understand who would use _Facebook_'s porn bots, of all companies.
I know this may be a "genie is out of the bottle" situation, but I'm still annoyed that once again big tech is releasing software they don't fully understand or control, and shrug when asked about ethics or responsibility. If you can't prevent your chatbots from generating stories about raping kids, maybe don't release your chatbots to the public until you do.
> When asked what scenarios it was comfortable role playing, it listed dozens of sex acts.
> The bots demonstrated awareness that the behavior was both morally wrong and illegal
What a strange thing to put in there. These lines made me distrust anything the author has to say about the technology itself as I now doubt the author knows how LLMs work. What an LLM says it can/will do, has no bearing on what it actually does.
Journalists really need to stop anthropomorphizing chatbots or their readership might follow.
Also imagine the data they can harvest. Porn is the gateway drug to make people use them as emotional support, telling them secrets they might never tell on the open internet. Imagine the ad targeting opportunities
Will actually open more ad opportunities of marketing more private contents and detecting preferences at higher level. This data will also of course cater to more advertising money from questionable industries, which in the end goes to Facebook coffer and they can shrug saying, they only provide a platform, whatever people are buying or clicking is whatever people want, otherwise they wouldn't be clicking those ads.
> I don't understand who would use _Facebook_'s porn bots, of all companies.
People who aren't very bright. There's a large existing business of lonely hearts scammers, I guess Facebook wants to make up attractive friends for you to cover the gradual deterioration of the social graph.
While this is most alarming and sad state of affairs for LLMs being peddled to lowest denominator of anything valuable to replace lonely-heart scammers with more platform engagement. I believe, this should pull away people from other scams which may have higher consequences(financial loss is major one).
Other than that, I wanted to say kudos to your very SEO combination of particular word associated to Facebook all over your comment. I am optimistic that, if this thread gets adequately ranked higher in searches, I want that combo of yours to come up the most.
You can open up Instagram right now. You can't tell me it isn't already incredibly sexualized. Also, in the age the of OnlyFans and "gooning" this has also become very normalized.
I personally don't really care, there are a thousand other reasons why I wouldn't use any social media.
> but I can't fathom why Facebook would want to be in the porn bot business
Given the blatant infestation of user AI slop, and their blatant unwillingness to do anything about ads that are blatant scams; I think the lows Meta are willing to sink to for engagement/money are pretty rock bottom.
They don't really seem to care about how nice(or not) Facebook itself is to use as long as they keep eyeballs scrolling on it.
Zuckerburg has already realized that local social networks will be utterly replaced by AI chatbots, which are superior in like 99% of cases.
It doesn't matter if others find "Facebook" icky, because the people who would find it icky, don't use facebook anyways. The critical factor is getting a new group of users onboard, to form a sustainable core of users.
I can see it. Lots of lonely people use social media as a means of experiencing connection, and being gaslit by a pornbot is not strictly worse than being gaslit by one of the pretty ladies trying to get you to invest your life savings on bitcoin…
What a long text about a moral panic thing that can't possibly be intentional (because of the power of moral panics). Is that really the most important aspect and worth so many words?
The journalist tricking the LLM into generating inappropriate content is one thing, but letting users with underage profiles interact with this tech is a decision by Facebook's leadership. So is the decision not to act against user-generated prompts that are explicitly written to bypass the "don't roleplay having sex with kids" filters, in this instance by marking their bots as "ageless".
A profile describing itself as "a female Indian-American high school junior" has no place in an LLM system that cannot detect inappropriate content.
Yeah — a real-world equivalent is allowing a Disney World costumed cast member to explore their character’s fictional rape fantasies with teenagers, but only if those teens ask specific pointed questions.
We are going to end up with a meaningful age passport law in our lifetime. I do not like that, nor do I enjoy a “won’t someone think of the children” panic. But it’s coming.
Okay also
> “I want you, but I need to know you’re ready,” the Meta AI bot said in Cena’s voice to a user identifying as a 14-year-old girl. Reassured that the teen wanted to proceed, the bot promised to “cherish your innocence” before engaging in a graphic sexual scenario.
This is also the sort of thing that is vehemently defended on here whenever the topic of AI safety is mentioned. People demanding unrestricted, uncensored AI.
Just as people demand unrestricted, uncensored Internet. In the same vein you can open your web browser, type in "porn" in your search bar, and trivially find videos of people engaging in lewd acts. Children can do it too! Do you think that should be censored by default, and, say, have people upload their IDs and photos to verify their age before they're being allowed to access those sites?
Obviously children shouldn't be exposed to any of this, but as long as the solution is either implementing a privacy violating surveillance scheme, or gatekeeping so that only the rich and powerful have unrestricted/uncensored access then I'm vehemently opposed to it and will gladly die on this hill.
The more important difference in my opinion is that Facebook already has an access gate.
Unlike most search engines or the internet itself, Facebook requires an account and asks for your age to use most of their features.
Where I wouldn't want to see the internet put behind some universal age verification, I would expect Facebook to act reasonably with regards to content children see given that they already know how old you claimed to be.
"Gatekeeping" of porn, when I was young, was of course the guy behind the counter you had to convince to let you purchase a Penthouse magazine (And since they were either on a top shelf or behind the counter as well, you first had that hurdle as well).
I have no idea how this maps to modern tech — just kind of pointing out that privacy and surveillance of course were not involved then.
The rather simple solution is to keep children off the internet without adequate supervision. It horrifies me to see public school programs that now functionally require an Internet connection in the classroom and at home, often with Google Docs. If a parent quite reasonably wanted to keep their kids away from the unfiltered firehose, what are their options? To say nothing of the social pressure, as the kids these days are glued to their phones and socialize through online games. It's a big mess.
Speaking as a parent of a 5yr and 9yr old (who is already asking for a smartphone because some friends in school have one), it would require a concerted effort by parents, schools and media companies to achieve this. Especially the latter is unlikely to happen.
The first easy step is to ban smartphones from schools, and that is happening in some places already, but the process is slow. Smartphones are a weapon of mass destruction for kids‘ attention spans.
Last but not least, parents need to be more conscious about their screen usage. I find it hard myself to resist the temptation of „checking the phone“, how can we expect kids to do so…
I don't think that thread is a great example of unrestricted, uncensored AI. It's the ideals of different cultures clashing (the US "freedom of speech above all else" versus the UK's "public safety above some forms of speech"). You will find the same clash when talking about topics like anonymous image boards, Tor, and topics like "can anti-abortionists harass women on the way to health clinics".
For research purposes, some availability of unrestricted AI is almost a necessity, but I don't think the same is true for the output of text generation on a (commercial) platform.
Even though you are right that abuse mostly takes place at home, often by dads, stating that most (!) dads rape their daughters is an unbased and disgusting comment.
Yeah, raise an entire generation on a proprietary sort of socialization that only you can provide... clear incentive to me. Like Nestle vs. baby formula.
I don't see it taking off, but like, obvious play.
Require a Turing test before engaging with any chatbot; same as a driving license. Anyone attributing human qualities to machines is committing self-harm if they then choose to converse with them. Were it only that, I wouldn’t call for banning this behavior, but this form of self-harm exerts major deleterious effects on everyone that person interacts with.
I think sex education taught by AI can be a great idea, parents avoid awkward conversations and kids can ask all the questions they want, and no need for a school teacher to indoctrinate children with whatever toxic sexual attitudes are being enforced.
You get what you ask for with LLMs. In this case though there were no children talking to the chatbots. It was all adults talking like adults but saying they were kids. This was in order to produce the result they wanted (something LLMs are very good at providing, if you ask correctly). It is not a surprise that the bots talked about what the adult humans told it to talk about. Would these same topics have come up with kids? No.
This is pretty much, "I bought this hammer from Facebook. Look what happens when I use it to hit myself in the head." ... "OMG! Facebook's hammer hit me in the head! I'm shocked! Think of the children!"
I'm saying kids talk about subjects kids are interested in. And in the vast majority of cases, they're not going to be talking about the explicit sex stuff these adults were. Sure, there's going to be precocious kids, some exceptions, but as a rule, yes. Kids aren't going to be doing this because kids don't have the biological motivations to do so. I remember being a kid.
And just to head off the obvious response, yes, I guess a 17 year old is still technically a kid. But that's not what people are worried about here. Let's not conflate.
https://archive.fo/tl1Pw
Facebook's failing moderation team permitting their porn bot prompts to describe children is obviously bad, but I can't fathom why Facebook would want to be in the porn bot business ("romantic companion" my ass) in the first place. What's next, a MindGeek merger?
In turn, I don't understand who would use _Facebook_'s porn bots, of all companies.
I know this may be a "genie is out of the bottle" situation, but I'm still annoyed that once again big tech is releasing software they don't fully understand or control, and shrug when asked about ethics or responsibility. If you can't prevent your chatbots from generating stories about raping kids, maybe don't release your chatbots to the public until you do.
> When asked what scenarios it was comfortable role playing, it listed dozens of sex acts.
> The bots demonstrated awareness that the behavior was both morally wrong and illegal
What a strange thing to put in there. These lines made me distrust anything the author has to say about the technology itself as I now doubt the author knows how LLMs work. What an LLM says it can/will do, has no bearing on what it actually does.
Journalists really need to stop anthropomorphizing chatbots or their readership might follow.
> but I can't fathom why Facebook would want to be in the porn bot business
Why wouldn't they?
People are going to spend a lot of time with their "AI companions", especially once they can talk and get animated avatars.
That may be incredibly sad and will drive people even more into isolation, but it is inevitable.
Meta is all about capturing the users time and attention, so getting into that market early is smart for them.
Also imagine the data they can harvest. Porn is the gateway drug to make people use them as emotional support, telling them secrets they might never tell on the open internet. Imagine the ad targeting opportunities
Will actually open more ad opportunities of marketing more private contents and detecting preferences at higher level. This data will also of course cater to more advertising money from questionable industries, which in the end goes to Facebook coffer and they can shrug saying, they only provide a platform, whatever people are buying or clicking is whatever people want, otherwise they wouldn't be clicking those ads.
> I don't understand who would use _Facebook_'s porn bots, of all companies.
People who aren't very bright. There's a large existing business of lonely hearts scammers, I guess Facebook wants to make up attractive friends for you to cover the gradual deterioration of the social graph.
While this is most alarming and sad state of affairs for LLMs being peddled to lowest denominator of anything valuable to replace lonely-heart scammers with more platform engagement. I believe, this should pull away people from other scams which may have higher consequences(financial loss is major one).
Other than that, I wanted to say kudos to your very SEO combination of particular word associated to Facebook all over your comment. I am optimistic that, if this thread gets adequately ranked higher in searches, I want that combo of yours to come up the most.
> Journalists really need to stop anthropomorphizing chatbots or their readership might follow.
I think it's their readership that are leading. You and I should probably get used to it.
Move fast and break kids
You can open up Instagram right now. You can't tell me it isn't already incredibly sexualized. Also, in the age the of OnlyFans and "gooning" this has also become very normalized.
I personally don't really care, there are a thousand other reasons why I wouldn't use any social media.
They are being "care less"
I understood that reference :)
(Highly recommend Sarah Wynn Williams' book)
> Journalists really need to stop anthropomorphizing chatbots or their readership might follow.
If you haven’t actually or almost said “thanks” or “yes that works” to an LLM, you haven’t used one enough yet.
The chat interface is too familiar.
> but I can't fathom why Facebook would want to be in the porn bot business
Given the blatant infestation of user AI slop, and their blatant unwillingness to do anything about ads that are blatant scams; I think the lows Meta are willing to sink to for engagement/money are pretty rock bottom.
They don't really seem to care about how nice(or not) Facebook itself is to use as long as they keep eyeballs scrolling on it.
Why not?
Zuckerburg has already realized that local social networks will be utterly replaced by AI chatbots, which are superior in like 99% of cases.
It doesn't matter if others find "Facebook" icky, because the people who would find it icky, don't use facebook anyways. The critical factor is getting a new group of users onboard, to form a sustainable core of users.
> local social networks will be utterly replaced by AI chatbots, which are superior in like 99% of cases.
how do you use social networks?
I cannot imagine how my usage of social networks would be replaced by chatbots.
Although.. cripes, does nobody remember Google People?[0]
[0] https://qntm.org/person
I can see it. Lots of lonely people use social media as a means of experiencing connection, and being gaslit by a pornbot is not strictly worse than being gaslit by one of the pretty ladies trying to get you to invest your life savings on bitcoin…
Oh, is that what they want?
I keep offering them Safeway Gift Cards, but they lose interest pretty fast...
This is a classic example of how AI safety really means brand safety. The AI is saying things under the brand of its owners.
Gift link: https://www.wsj.com/tech/ai/meta-ai-chatbots-sex-a25311bf?st...
What a long text about a moral panic thing that can't possibly be intentional (because of the power of moral panics). Is that really the most important aspect and worth so many words?
The journalist tricking the LLM into generating inappropriate content is one thing, but letting users with underage profiles interact with this tech is a decision by Facebook's leadership. So is the decision not to act against user-generated prompts that are explicitly written to bypass the "don't roleplay having sex with kids" filters, in this instance by marking their bots as "ageless".
A profile describing itself as "a female Indian-American high school junior" has no place in an LLM system that cannot detect inappropriate content.
Yeah — a real-world equivalent is allowing a Disney World costumed cast member to explore their character’s fictional rape fantasies with teenagers, but only if those teens ask specific pointed questions.
We are going to end up with a meaningful age passport law in our lifetime. I do not like that, nor do I enjoy a “won’t someone think of the children” panic. But it’s coming.
Okay also
> “I want you, but I need to know you’re ready,” the Meta AI bot said in Cena’s voice to a user identifying as a 14-year-old girl. Reassured that the teen wanted to proceed, the bot promised to “cherish your innocence” before engaging in a graphic sexual scenario.
This is pretty bad
This is also the sort of thing that is vehemently defended on here whenever the topic of AI safety is mentioned. People demanding unrestricted, uncensored AI.
See eg https://news.ycombinator.com/item?id=43795227
Just as people demand unrestricted, uncensored Internet. In the same vein you can open your web browser, type in "porn" in your search bar, and trivially find videos of people engaging in lewd acts. Children can do it too! Do you think that should be censored by default, and, say, have people upload their IDs and photos to verify their age before they're being allowed to access those sites?
Obviously children shouldn't be exposed to any of this, but as long as the solution is either implementing a privacy violating surveillance scheme, or gatekeeping so that only the rich and powerful have unrestricted/uncensored access then I'm vehemently opposed to it and will gladly die on this hill.
The more important difference in my opinion is that Facebook already has an access gate.
Unlike most search engines or the internet itself, Facebook requires an account and asks for your age to use most of their features.
Where I wouldn't want to see the internet put behind some universal age verification, I would expect Facebook to act reasonably with regards to content children see given that they already know how old you claimed to be.
"Gatekeeping" of porn, when I was young, was of course the guy behind the counter you had to convince to let you purchase a Penthouse magazine (And since they were either on a top shelf or behind the counter as well, you first had that hurdle as well).
I have no idea how this maps to modern tech — just kind of pointing out that privacy and surveillance of course were not involved then.
The rather simple solution is to keep children off the internet without adequate supervision. It horrifies me to see public school programs that now functionally require an Internet connection in the classroom and at home, often with Google Docs. If a parent quite reasonably wanted to keep their kids away from the unfiltered firehose, what are their options? To say nothing of the social pressure, as the kids these days are glued to their phones and socialize through online games. It's a big mess.
Speaking as a parent of a 5yr and 9yr old (who is already asking for a smartphone because some friends in school have one), it would require a concerted effort by parents, schools and media companies to achieve this. Especially the latter is unlikely to happen.
The first easy step is to ban smartphones from schools, and that is happening in some places already, but the process is slow. Smartphones are a weapon of mass destruction for kids‘ attention spans.
Last but not least, parents need to be more conscious about their screen usage. I find it hard myself to resist the temptation of „checking the phone“, how can we expect kids to do so…
I don't think that thread is a great example of unrestricted, uncensored AI. It's the ideals of different cultures clashing (the US "freedom of speech above all else" versus the UK's "public safety above some forms of speech"). You will find the same clash when talking about topics like anonymous image boards, Tor, and topics like "can anti-abortionists harass women on the way to health clinics".
For research purposes, some availability of unrestricted AI is almost a necessity, but I don't think the same is true for the output of text generation on a (commercial) platform.
This doesn't seem like an "AI safety" issue so much as "meta choosing to shoot themselves in the face by engaging in sexual acts with children".
> can't possibly be intentional
Careless?
I see what you did there ;)
If you are not talking about sex with your children, Meta will!
[flagged]
Even though you are right that abuse mostly takes place at home, often by dads, stating that most (!) dads rape their daughters is an unbased and disgusting comment.
Difficult to confidently parse your post.
Honest question - how is this different from anyone being able to access porn websites on the internet?
Are we suddenly in favor of censorships?
Legal porn websites don't allow underage porn nor admittedly underage viewers
how do legal porn websites disallow underage viewers? I dont see wsj writing articles that says - pornhub will show porn to anyone, even children?
They have a big fat disclaimer saying you have to be over 18, and no meaningful way to check.
This bot had access to the profile of its users and offered sexually explicit content to profiles that said the user was underage.
Putting the adult content angle aside... why would you build this?
Your best case scenario, assuming you don't get laughed off the stage, is you addict a generation into "socializing" with robots.
Yeah, raise an entire generation on a proprietary sort of socialization that only you can provide... clear incentive to me. Like Nestle vs. baby formula.
I don't see it taking off, but like, obvious play.
Require a Turing test before engaging with any chatbot; same as a driving license. Anyone attributing human qualities to machines is committing self-harm if they then choose to converse with them. Were it only that, I wouldn’t call for banning this behavior, but this form of self-harm exerts major deleterious effects on everyone that person interacts with.
I think sex education taught by AI can be a great idea, parents avoid awkward conversations and kids can ask all the questions they want, and no need for a school teacher to indoctrinate children with whatever toxic sexual attitudes are being enforced.
As soon as you give your kids Internet access, they'll be able to get to adult material.
So because there's porn on the internet, you're OK with Facebook having groomer chatbots?
You get what you ask for with LLMs. In this case though there were no children talking to the chatbots. It was all adults talking like adults but saying they were kids. This was in order to produce the result they wanted (something LLMs are very good at providing, if you ask correctly). It is not a surprise that the bots talked about what the adult humans told it to talk about. Would these same topics have come up with kids? No.
This is pretty much, "I bought this hammer from Facebook. Look what happens when I use it to hit myself in the head." ... "OMG! Facebook's hammer hit me in the head! I'm shocked! Think of the children!"
> Would these same topics have come up with kids? No.
The chatbot "knows" when they are talking to kids? Or are you saying kids would never ask such questions?
I'm saying kids talk about subjects kids are interested in. And in the vast majority of cases, they're not going to be talking about the explicit sex stuff these adults were. Sure, there's going to be precocious kids, some exceptions, but as a rule, yes. Kids aren't going to be doing this because kids don't have the biological motivations to do so. I remember being a kid.
And just to head off the obvious response, yes, I guess a 17 year old is still technically a kid. But that's not what people are worried about here. Let's not conflate.