Ok, yes, we are building a dating app and yes it has everything you would expect to allow users to report content and users to our admins in the event they are being harassed, spammed, or just generally done talking to a particular person. But we decided very early on that would not be enough.
The dating market is pretty full and each app out there has it’s own well deserved reputation. The last thing we want when we launch is to open the door for a small number of trolls to make Challengr’s reputation ‘the app with all the dick pics’.
So we took action. As a tiny company, there is no way we would be able to manually look at every picture or video that gets uploaded to the app. More importantly, if we were getting a lot of nudity, we would not want to sear our eyes with what the general public would be submitting. So here comes the first step of Machine Learning into our platform, the dick detector AKA booby bounty.
We wont get too technical for Part I. If you want to see how we developed this bit, check out the follow on posts (Tech: Image Moderation and Tech: Video Moderation). But what we will get into here is how our rules should apply in this world.
The first question we asked was, how much is this all going to cost us and is it worth it? The answer to that comes pretty easy thanks to Amazon’s pay-as-you-go pricing sheet. It will cost us 10 cents a minute for each video we process and another 10 cents for every 100 images. That’s pretty reasonable, and as a brand new startup, essentially makes it free until we are pulling in some real usage.
The next question to answer: What does this AI actually tell us anyway? How ‘smart’ is it? It turns out it wont give us a name, address, social security number, nor the deepest and darkest secrets of the person being processed, but it does give us what we need, ‘this is nudity’ and ‘this is not nudity’. More specifically, these are the available results:
Graphic Male Nudity
Graphic Female Nudity
Female Swimwear or Underwear
Male Swimwear or Underwear
Now after reading this list you may asking, ‘Hmm… should we be filtering out Female Nudity?’ You don’t really hear too many complaints about boob pics being sent over the net. In fact, if we think guys don’t mind seeing some naked women, perhaps we can even save a couple bucks and not even bother processing any photos or videos uploaded by women at all!
Here at Challengr we try not to put our users in gender buckets like that and in the interest of equality and not making assumptions on what kind of experience our users want, we decided against that approach. But, you were on to something about saving a couple bucks, so we put a setting on everyone’s profile and just ask them: do you want us to filter your content for you? We default that setting to ‘yes’ and leave it at that. After we launch we’ll let you know what percentage of people open the door to some private pics.
Now the next big question. How do we deal with the people sending these graphic images. The answer to this one took a little bit of thinking
Option 1: Stop the upload in its tracks and just mark the post as a failure.
On its face, this seems like it would be an attractive option. There is no mistaking what happened, we show a little ‘Oops!’ message to the user uploading letting them know that kind of content isn’t allowed on the platform, and the person being sent the image never sees it.
So what’s the catch? Well, this is a person who clearly has a need to send some untoward content and if we tell them we blocked it, will start trying to get around it. For whatever reason, they will have a need to get their privates through the door and no matter how good our computer is, they will probably find a way. The next drawback to this approach is a false positive. What if they upload a totally innocent picture and we tell them its porn? That would rub our users the wrong way.
Already this option is looking pretty rough but the one that knocks this one out of contention is: That’s way too much work! We would have to build out an entire error flow for users to be notified that their penis was blocked, allow for an interface for them to dispute the block or allow them to accept it and delete it from their profile. And if they dispute, would need to have to have a real person review and then have a pipeline to restore override the block, and then notify the uploader that ‘sorry our bad, we sent it through ok’.
Option 2: Shadow Moderate
In this scenario, we are going to make the content moderation completely invisible to our users, both the one sending the nudity and the one potentially receiving it. When a private picture gets sent up, we create the post as normal but with a special status of ‘processing’. When looking at your own posts, we simply ignore the ‘processing’ state and return it as normal, so you think it went through just fine. For everyone else, if the state is not ‘complete’ it gets filtered from their feeds.
After we check for nudity, we change the state to either ‘moderated’ or ‘complete’. If we mark it complete, we send a quick notification to the receiving user that a safe piece of content was sent to them and they should check it out. If we mark it ‘moderated’ we don’t have to do anything, it just exists on the platform and the only one who can see it is the creator of the post.
So now, any piece of inappropriate content uploaded looks like it works and the uploader never knows if we caught it or not and therefore wont try to game the system. If we mess up and it is a false positive, no one will ever know! It will be as though the post was just not good enough for the receiver to mark it as a match.
So naturally, we went with option 2.