Sciencemadness Discussion Board - Tired of reporting spam

XMB hasn't been actively developed since 2009, and I'm pretty sure we had a consensus a long time ago that we're going to have to transition to new software if we ever want to address this problem. Correct me if I'm wrong.

I think xmb will work fine with changes. But not up to me... I'm a "lazy" software developer, I hate to do things that are already "solved" or good enough. Vulnerabilities would be a credible reason to change the platform IMO (e.g. a former workplace had ISIS photographs/crap put on the homepage... that was interesting). Hopefully this website's data is backed up somewhere just-in-case.

My suggestion, as someone with "trusted" credentials can do, is set up your own "bot" to report the other bots... Use either Selenium or Sikuli:

Selenium: If on linux, use either the chromium-driver or firefoxdriver packages, and the python-selenium package. Selenium automates browsers, and uses their APIs to parse things.
Sikuli: An alternate approach is to do everything visually (no browser based API), using software like Sikuli. Sikuli uses Java... it is cross-platform.

I've used Sikuli a lot before... I can write a spam report script for it if you ask. But Sikuli has to use optical character recognition to understand text (error prone), while Selenium handles text perfectly. Less effort though to just update this forum's software with one of Streety's solutions though...

Either can be run on linux in its own X11 server 24/7, so as not to clutter up your main desktop... but computer would have to be on 24/7, or run it on the "cloud" at some interval...

Using either approach... every ~5-10 minutes... refresh 'Today's Posts', parse new posts, and report new users with 2+ posts on the same day containing links. Give it both a "trusted" user and admin/mod's credentials (script could log in/out of each) and you'd wipe out classified spam.

Quote: Originally posted by WGTR

Now, since I have a few dozen sent messages that I'd like to keep, I have to go into my outbox and manually look through all 5,600 of these spam reports so that I don't accidentally delete something important. Sigh.

Similarly a script could be made to do this for you, parsing each u2u message. i.e. "IF spam report, THEN delete." If you attach a screenshot of one of these spam reports or better forward a bunch to me u2u EDIT: Nevermind! I have them in my outbox too

I'll attach it as a zip. You can watch it as it moves the mouse.

Sikuli is pretty great (but text parsing is imperfect, hard to get right).

EDIT2: I've attached a Sikuli script you can use to delete all those spam reports in your u2u outbox/inbox. Should go in ~/.Sikulix/scripts I'm using v1.1.0, but it should work for v1.1.4 too. If you have no status bar on your browser (chrome), it may skip over the last item every screen. If you have different fonts or screen size than me you may have to change the images/regions.

Attachment: scimadness_u2u.sikuli.zip (4kB)
This file has been downloaded 442 times

[Edited on 25-9-2018 by andy1988]

Melgar - 25-9-2018 at 07:15

I'd like an opportunity to back up my private messages before any migration. Actually, I'd like an opportunity to back them up anyway. I haven't spotted a way of doing that.

First of all, what format would you want your private messages in? CSV? HTML with minimal formatting? Plain text?

During any migration, there would be a period of time where both sites would exist simultaneously, one running phpBB and one running XMB. During that time period, we can all have the chance to audit the new site and make sure everything looks right. And we'd keep a backup of the server before taking it down, for certain.

Also, since phpBB has a separate system of reporting spam that doesn't use U2U messages, I should probably have my back-end scripts clear out all the spam reports that are in people's inboxes and outboxes.

As for automated GUI-based scripting software, my go-to has always been AutoHotkey. It's free and open-source, and works quite beautifully with Windows. I've even written scripts with it to grind my skills in MMORPG games back when I used to play them. Still, those are all half-measures and can only do so much to stem the rising tide of spam.

Right now, I set the maximum attachment size per file to 8MB, to match that here. For larger files, I can set up an FTP server that uses our forum passwords, and probably make it so an admin has to enable FTP access for users.

I've been trying to think of custom bbcodes that might be useful for chemistry too. Maybe something that can automatically work out the molar mass of a substance via its molecular formula, or by scraping the data from PubChem and storing it locally if it doesn't recognize it? Maybe link to the Wiki page if one exists, and have that also show common abbreviations for chemicals that can be used to reference their properties on the forums? This is all backend SQL stuff, which is my forté, so it'd actually be easier for me to do than creating a style or writing PHP code.

unionised - 25-9-2018 at 10:03

If there's anything I can to to help get rid of the spammers, let me know.

streety - 25-9-2018 at 15:46

Quote: Originally posted by symboom

Anyone know if they are using AI (artificial intelligence) yet
Got to be ahead of the game. Divergent and convergent thinking

Open message
To streety
This guy lesterpq11 look at for pattern see key words to outwit the AI

AI seems to try to imitate humans where ever it gets the info from

[Edited on 22-9-2018 by symboom]

A well designed AI model could do far better than what we are seeing. The approach being taken is definitely quantity over quality.

If the goal was for posts to stay on a site and someone was involved who knew what they were doing there have been some groundbreaking advancements creating fakes almost indistinguishable from the real thing. If you are interested in this topic take a look at generative adversarial networks. This would be just the start.

I'll take a look at lesterpq11.

Quote: Originally posted by RogueRose

This may have been asked and stated but I haven't seen it. How many reports of spam are needed for all the users posts to be deleted?

What I want to know is if I see 10 posts by a user, how many need to be reported for the program to sweep up all their posts?

This is the most complete description I know.

Quote: Originally posted by Polverone

I have tweaked the reporting process, somewhat inspired by violet sin. Before spam had to be reported by two trusted members or one moderator before it was auto-deleted. Now it will be auto-deleted if it is reported by one trusted member and one 'semi-trusted' member. Everyone who has been registered at least 60 days and has at least 50 posts automatically becomes a member of the 'semi-trusted' group. This should reduce the waiting time for spam deletion, and means that any frequent user of the site can help stamp out spam.

The recommendation seems to be to report the thread with the most views already.

Quote: Originally posted by CuReUS

My own suggestions-
1.Most spammers have a link in their message,so we could detect posts with links and block them(for 1st post only)
2.Do not allow newly registered members to post more than 1 message or in more than 1 sub forum.
3.Block usernames or posts with non english alphabets

someone had posted an amazing idea is this thread,but I can't seem to find that post now.The idea was to run usernames through a password strength checker.Since bots use long alphanumerical strings,they would indirectly make very strong passwords,which could be detected and blocked.
We must do something fast,or pretty soon we would have to build another arc to escape this flood

[Edited on 23-9-2018 by CuReUS]

Currently the limiting issue is getting involvement from Polverone. The work-around is running a script off the server to clean up any spam soon after it's posted.

I've written a script that woelen is now able to run and the percentage of non-english characters is an important feature in the machine learning model for spam prediction.

The password strength metric on username idea is not one I've seen before and sounds interesting. I'll give that a try.

Quote: Originally posted by RogueRose

I have a feeling that there is more going on than just spam. I have a feeling that the posts might be a way to pass messages to others with no record of them. The posts are up for a short time and then the "system" erases them. While they are up they are grabbed. I would suspect that the spam bots wouldn't continue to post here if it wasn't getting some kind of return. It wouldn't post with such furry unless there was a benefit being had and I don't think it is members buying access to adult sites or ED pills.

This could be a serious issue that really needs taken care of and is wreckless allowing it to continue.

This is an interesting idea I had not considered. I agree with others in thinking it is unlikely but I agree with you in thinking it would be virtually impossible to detect. Ultimately though our situation is the same, the forum receives posts we don't want.

I think Melgar touched on this but I want to discuss your comment on "members buying access to adult sites or ED pills". This would be the primary effect but I don't think this is where the money is. Instead it is more likely to be in the secondary or tertiary effects.

1. People see the spam on this forum and click through to buy the thing/have their computer highjacked by viruses etc
2. Across the thousands of forums these spam programs are run against someone sees the spam and clicks through to buy the thing/have their computer highjacked by viruses etc
3. People keep the idea alive that money can be made by spamming forums so they can sell forum spamming software to the people they are able to deceive.

Quote: Originally posted by andy1988

Selenium: If on linux, use either the chromium-driver or firefoxdriver packages, and the python-selenium package. Selenium automates browsers, and uses their APIs to parse things.
Sikuli: An alternate approach is to do everything visually (no browser based API), using software like Sikuli. Sikuli uses Java... it is cross-platform.

Way ahead of you.

The script has been written, woelen has run it to test, and it just needs to be set on automatic.

Switching to phpBB isn't going to be an end to the spam. It will make the site easier to maintain in future but other steps will be needed to prevent spam beyond the default. It is also a longer term change. We have no idea how long it will be before Polverone has time to work on the site.

Currently we have people with the power to make changes but no time/interest and people with time/interest but no power.

Melgar - 25-9-2018 at 21:21

Currently we have people with the power to make changes but no time/interest and people with time/interest but no power.

Yes, I do believe this gets at the heart of the problem.

Have you looked at my phpBB implementation recently? I've made some decent progress lately. If you register there, I can make you an admin, or if you send me a public key, I can add you to authorized_keys. Spam can definitely be reduced from a flood to a drip by switching forum software, not to mention, Polverone and others have been calling for it for years now.

[edit]
Rogue pics removed.

[Edited on 26-9-2018 by j_sum1]

j_sum1 - 25-9-2018 at 22:08

Heh. Nice pics, Melgar.
I think they migrated from another thread.
I'll remove them for you.

streety - 26-9-2018 at 04:55

I've not taken a good look at the progress you've made yet. I was out of town over the weekend, actually up in NYC, and just now getting back on top of things. I'll take a good look this evening, sign up etc.

Are you sure about the spam? I would have thought that the default phpBB install would be more susceptible to spam. I thought it would be the extensions that exist due to its ongoing development and community and the ease we can add additional countermeasures that would make the difference.

streety - 26-9-2018 at 17:23

I found all the posts created by the account lesterpq11 and ran them through the spam classification model. There were a few close to 0.6 that I'm not too happy about but nothing too challenging. The model will eventually need to be updated as the spam changes but for the moment it should be fine.

Code:


Topic                                   |Post date         |Spam score
----------------------------------------+------------------+----------
Matured site                            |2018-09-21 21:25  |0.947
Delivered adult galleries               |2018-09-21 21:25  |0.733
Sexual pictures                         |2018-09-21 21:24  |0.932
Communal pictures                       |2018-09-21 21:24  |0.867
My unfamiliar website                   |2018-09-21 21:23  |0.783
Matured purlieus                        |2018-09-21 21:23  |0.920
Pictures from community networks        |2018-09-21 21:22  |0.845
Adult galleries                         |2018-09-21 21:21  |0.911
Grown up galleries                      |2018-09-21 21:21  |0.663
Social pictures                         |2018-09-21 21:20  |0.760
Loose galleries                         |2018-09-21 21:20  |0.898
Matured site                            |2018-09-21 21:19  |0.688
Pictures from collective networks       |2018-09-21 21:18  |0.840
Communal pictures                       |2018-09-21 21:18  |0.688

I ran all the usernames through this password strength checker and plotted the histograms for spam accounts, and 0, 1-24, and 25+ post accounts.

username_as_password_strength.png - 86kB

There is a lot of overlap so I doubt much could be done with this. It's possible greater separation could be achieved by limiting the types of permutations considered.

fusso - 27-9-2018 at 05:39

What pic did Melgar post?

How's the password strength calculated?

j_sum1 - 27-9-2018 at 05:58

What pic did Melgar post?

Not that it matters much:
https://www.sciencemadness.org/whisper/viewthread.php?tid=93...

How the pictures got appended to another post, I don't know. But it has happened before.

CuReUS - 27-9-2018 at 06:26

There is a lot of overlap so I doubt much could be done with this

Maybe we could tell future members(while they are signing up) to not use complicated usernames and instead keep them short and simple ?

streety - 27-9-2018 at 09:00

The password strength is calculated using the zxcvbn package. A very thorough description is at https://blogs.dropbox.com/tech/2012/04/zxcvbn-realistic-pass...

Asking new members to keep their usernames short and simple might not be well received. Many people have a username they re-use in multiple places and identify strongly with.

Melgar - 27-9-2018 at 18:51

I've not taken a good look at the progress you've made yet. I was out of town over the weekend, actually up in NYC, and just now getting back on top of things. I'll take a good look this evening, sign up etc.

Are you sure about the spam? I would have thought that the default phpBB install would be more susceptible to spam. I thought it would be the extensions that exist due to its ongoing development and community and the ease we can add additional countermeasures that would make the difference.

I'm using the most recent version of phpBB now. Actually, a new version came out just last week. Once I figured out the database structure, having the better documentation for the newer versions ended up being more useful to me for figuring out how to make everything work. The main reason that XMB is so susceptible to spam is that there hasn't been a new version released in ~5 years or so, whereas phpBB has at least kept up with the pace of spambot evolution.

Anyway, I noticed you registered at the test site I set up, so I made you an admin. Feel free to poke around. Here's the test site again, for anyone curious:

http://35.185.63.230/talk/index.php

The style was chosen more or less randomly, and can easily be changed at any time. I've been focused a lot more on making sure that content is preserved.

And yeah, I'm not sure why those pictures were duplicated from my previous post. Just another weird glitch with this software, I guess.

[Edited on 9/28/18 by Melgar]

fusso - 28-9-2018 at 12:04

Isn't "ScienceMadness" a single word?

Why we have fricking lots of spam today???

fusso - 6-10-2018 at 12:40

Why we have fricking lots of spam today???

phlogiston - 6-10-2018 at 13:54

Someone here said it before, I forgot who, but it is interesting that most spam posts contain incomprehensible crap, just strings of characters that do not make up words in any language.
The effort put into posting them suggests they serve some kind of purpose.
Apparently they contain information that is useful to someone, and our forum provides a means of transmitting it from A to B. Even if the posts exist only briefly before being deleted.

Regardless, it is becoming extremely annoying.
I find myself just giving up sometimes. 9 out of 10 times when I actually bother to log in, I do so only to report spam, rather than to enjoy discussing mad science with like-minded souls.

Assured Fish - 6-10-2018 at 14:58

I dont think anyone is manually posting that spam, or creating the accounts, i suspect its all automated and the SM system is just rather easy to do that with.

I think we need to set up a bot verification system upon logging in.
At the very least something when creating an account, i think just about every site on the net has one nowadays, we are very very very behind the times.

Melgar - 6-10-2018 at 15:01

Quote: Originally posted by phlogiston

The effort put into posting them suggests they serve some kind of purpose.
Apparently they contain information that is useful to someone, and our forum provides a means of transmitting it from A to B. Even if the posts exist only briefly before being deleted.

The effort at making theses posts is zero. Or at least, it's so little that it's basically nothing. It's the exact same amount of effort as sending a spam email. A lot of these programs are poorly-written by programmers in India and China and Russia.

Have you ever seen the movie "A Beautiful Mind"? Where he believes that there are hidden messages in the newspapers, and he has to decode them in order to stop some communist plot? Down that road, madness lies. Literally. And not the good kind of madness.

j_sum1 - 6-10-2018 at 15:17

I have just done a bit of a clean-up. I noticed it building before I went to bed last night but didn't have time to attend to it. My apologies.
Anyway, 118 spam posts gone. More than our reporting system was able to handle. (I don't know why some spam posts persist even after being reported.)
I am now about to check my inbox to see if theer is anything in there other than spam reports.

Thanks for sticking in there. I am really looking forward to a better solution than this.

Melgar - 6-10-2018 at 18:18

Hey, I just had an idea. You know the saying "use a thief to catch a thief"? Well, how about using a bot to catch bots? If someone gave me moderator privileges, I could set up my old laptop with a macro on it that'd refresh the boards like every ten seconds and delete all the posts from obvious bots.

Or Polverone could implement that fix for the user registration page. That would work better, I imagine, but if that's not happening, an auto-moderator could help with this. I could also change the rules it uses to identify bots, which would then mean that some of the ideas that have been floated could be put to the test.

Edit: I'm a bit worried that I'll completely fix the phpBB migration issues that are left, only to have Polverone be AWOL with no way of contacting him. I'm not sure to what extent I'd be trusted with forum data when we're ready to switch. I would say that I'm really terrible at being dishonest so I think you can trust me just fine. And I'm much more mature professionally than socially. I know it's weird to ask about these things, but there are certain systems that I can't test with the database that I have. Also, I quite like this site, and want it to continue to operate for as long as possible.

[Edited on 10/7/18 by Melgar]

Elrik - 6-10-2018 at 20:56

Quote: Originally posted by phlogiston

...Apparently they contain information that is useful to someone, and our forum provides a means of transmitting it from A to B. Even if the posts exist only briefly before being deleted...

Thats exactly right.
You'll notice the bot spam has urls in it. One way google calculates ranking in search results is how often it sees urls being posted on websites. The bot designer spams urls that make him money somehow, be it ads that show on those pages, scams to get peoples money, etc and when google indexes this site those urls go up in the google search rankings. This is a very old tactic.
Its naustalgic seeing a forum use software from the turn of the century, but human verification in registration should really be implemented

btw, is reporting spam even useful? I would think admins would have no problem spotting it. Threads about butyl caproate: good, threads about adult galleries or "ghsvakydfhb": bad

[Edited on 2018-10-7 by Elrik]

j_sum1 - 6-10-2018 at 23:09

If two established members report spam with the single word, "spam" then it activates a script to delete all posts/threads by that user and bans the account.
For most efficient use
1. Make sure you include the word, "spam"
2. Do not post anything in the thread
3. Take a look at the view count and try to target threads that are likely to have been reported already.
4. You only need to report one thread per username.

Admittedly the script is not always effective. There are a few known issues and possibly some unknown issues. But when it runs the spam is vaporised pretty quickly.

So, yes. Spam reporting does work. It is vastly preferable to leaving it there, logging off and coming back later. If everyone reported a couple of spammers per visit then the board would be a lot cleaner. As it is, yhere are probably only 20 or so members who are regularly reporting but I know there are plenty more frustrated with the problem.

Melgar - 7-10-2018 at 00:29

If two established members report spam with the single word, "spam" then it activates a script to delete all posts/threads by that user and bans the account.
For most efficient use
1. Make sure you include the word, "spam"
2. Do not post anything in the thread
3. Take a look at the view count and try to target threads that are likely to have been reported already.
4. You only need to report one thread per username.

Admittedly the script is not always effective. There are a few known issues and possibly some unknown issues. But when it runs the spam is vaporised pretty quickly.

So, yes. Spam reporting does work. It is vastly preferable to leaving it there, logging off and coming back later. If everyone reported a couple of spammers per visit then the board would be a lot cleaner. As it is, yhere are probably only 20 or so members who are regularly reporting but I know there are plenty more frustrated with the problem.

Is that what "TestAccount" is for?

Reporting spam feels like trying to empty out the Pacific Ocean with a spoon. Who wrote the script anyway? And in what language?

unionised - 7-10-2018 at 00:30

You forgot to mention how satisfying it is when you report a spammer and their stuff disappears.
Folk might lie to try it.

fusso - 7-10-2018 at 04:50

So, yes. Spam reporting does work. It is vastly preferable to leaving it there, logging off and coming back later. If everyone reported a couple of spammers per visit then the board would be a lot cleaner. As it is, yhere are probably only 20 or so members who are regularly reporting but I know there are plenty more frustrated with the problem.

That means spams won't go away if I report spams everyday while keeping myself logged in for a long time?

j_sum1 - 7-10-2018 at 05:50

@fusso
Sorry if i was unclear. Report spammers. Every bit helps.

streety - 7-10-2018 at 07:17

I thought there were 3 types of member the spam deletion script considered:

Admins/Mods
Trusted reporter - A hard coded list of members
Established members - a member for 6 months and 50 posts?

A spam report from a single admin/mod triggers deletion.

A spam report from both a trusted reporter and an established member can also trigger the deletion script.

I may be wrong but this was my understanding.

The script I wrote should escalate the spam report from a single member when the post has spam-like features. Hopefully this can be deployed soon. It does seem like the spam is getting worse.

[Edited on 7-10-2018 by streety] Yesterday was anomalous with spam sticking around for an unusually long time.

The trend is troubling though. Both the number of spam posts and the time taken to delete them is increasing.

spam_posts_durations_per_day_20181007.png - 130kB

[Edited on 7-10-2018 by streety]

fusso - 7-10-2018 at 08:16

Why can't we have the report function to also target the user alongside with individual posts (ie report the user directly, instead of his posts)?? :mad:

Edit: the spambot OFFERVEASSIDA has spammed almost 50 posts up till now and I think someone would be interested to investigate about it.

[Edited on 07/10/18 by fusso]

S.C. Wack - 7-10-2018 at 10:28

If two established members report spam with the single word, "spam" then it activates a script to delete all posts/threads by that user and bans the account.

So...anyone who quickly spams or sock puppets 100 posts can delete anyone they choose and all their posts, eh what?

fusso - 7-10-2018 at 10:58

Quote: Originally posted by S.C. Wack

If two established members report spam with the single word, "spam" then it activates a script to delete all posts/threads by that user and bans the account.

So...anyone who quickly spams or sock puppets 100 posts can delete anyone they choose and all their posts, eh what?

I dont't think any trusted members would want to delete another real member unless they want to revenge?

Heptylene - 7-10-2018 at 11:12

Quote: Originally posted by S.C. Wack

If two established members report spam with the single word, "spam" then it activates a script to delete all posts/threads by that user and bans the account.

So...anyone who quickly spams or sock puppets 100 posts can delete anyone they choose and all their posts, eh what?

I dont't think any trusted members would want to delete another real member unless they want to revenge?

No but a spammer quickly creating 2 accounts and spamming 100 posts with each account would give him the power to delete any account. Scarry, I'm not sure that's a good idea.

Apart from that, it gives me confidence that reporting spam actually is useful (at least when I'll have 100 posts. Gotta spam till I get there

)

Tsjerk - 7-10-2018 at 11:17

@ fusso, did you nominate yourself member of the day? Really?

S.C. Wack - 7-10-2018 at 11:24

The 100 I mentioned is 2x50. So there's a 6 months thing too? Long before the spam and even more so now I'm convinced there is at least 1 person here who could meet this standard even if it required 5 members, probably also responsible for the spam.

BTW I did this with OFFERVEASSIDA's oldest post and if I had any effect, apparently OFFERVEASSIDA reregistered immediately, or the script only deletes posts made before the report. EDIT: the situation is fluid...never mind.

[Edited on 7-10-2018 by S.C. Wack]

fusso - 7-10-2018 at 12:52

Quote: Originally posted by Heptylene

Quote: Originally posted by S.C. Wack

If two established members report spam with the single word, "spam" then it activates a script to delete all posts/threads by that user and bans the account.

So...anyone who quickly spams or sock puppets 100 posts can delete anyone they choose and all their posts, eh what?

I dont't think any trusted members would want to delete another real member unless they want to revenge?

No but a spammer quickly creating 2 accounts and spamming 100 posts with each account would give him the power to delete any account. Scarry, I'm not sure that's a good idea.

No, I mean only members in the 'trusted' group in Polverone's list, not 'semi'-trusted. Therefore spamming 100 posts won't give you such status.

fusso - 7-10-2018 at 12:59

@ fusso, did you nominate yourself member of the day? Really?

Yes... but a few days ago

And how come this OFFERVEASSIDA can still spam posts after all its posts deleted?!(ノ#`Д´)ノ彡┻━┻

[Edited on 07/10/18 by fusso]

streety - 7-10-2018 at 15:29

My understanding is that the automated deletion script has some hard limits. Some are deliberate and others are limitations.

I think it only works for new members (a day old?) so even if two members with trusted reporter status decided to go rogue they couldn't delete the account of anyone posting here. I think I'm the newest member and it would need to be a deliberate action by an admin.

The limitation is it can only handle certain types of posts. OFFERVEASSIDA both created new topics and also posted to existing topics which is more unusual. I think if a spam account posts to a topic and there are then posts from a legitimate member the automatic script is unable to delete it.

I hope my comment did not alarm anyone. The script seems to be quite conservative and the risk is, I think deliberately, weighted towards not deleting a spam member rather than erroneously deleting a real member.

fusso - 7-10-2018 at 15:52

My understanding is that the automated deletion script has some hard limits. Some are deliberate and others are limitations.

I think it only works for new members (a day old?) so even if two members with trusted reporter status decided to go rogue they couldn't delete the account of anyone posting here. I think I'm the newest member and it would need to be a deliberate action by an admin.

The limitation is it can only handle certain types of posts. OFFERVEASSIDA both created new topics and also posted to existing topics which is more unusual. I think if a spam account posts to a topic and there are then posts from a legitimate member the automatic script is unable to delete it.

I hope my comment did not alarm anyone. The script seems to be quite conservative and the risk is, I think deliberately, weighted towards not deleting a spam member rather than erroneously deleting a real member.

A better way is to move the reported posts to a hidden forum and have the mods decide whether that's spam or not. And make trusted members immune to being deleted completely.

unionised - 8-10-2018 at 11:14

I'm sure I'm not the only one wondering what's going on at the moment spam-wise.

Melgar - 8-10-2018 at 13:22

Well, woelen just gave me spam-fighting powers, part for fighting spam and part for the continuing board transition. I didn't know what I was getting myself into. Now every time I switch pages, I have a whole bunch of new U2U messages from posts that have been reported, and in most cases, that no longer exist.

But I'd had some ideas for finding and destroying spam in the works, so now I can actually implement them.

I know Kung Fu.

:cool:

Magpie - 8-10-2018 at 14:15

Would it be possible to have the forum software automatically delete:

1. posts in Russian?
2. posts with the word sex in the title?

fusso - 8-10-2018 at 14:21

Well, woelen just gave me spam-fighting powers, part for fighting spam and part for the continuing board transition. I didn't know what I was getting myself into. Now every time I switch pages, I have a whole bunch of new U2U messages from posts that have been reported, and in most cases, that no longer exist.

But I'd had some ideas for finding and destroying spam in the works, so now I can actually implement them.

I know Kung Fu.

:cool:

Wow congrats Melgar! Now you can help reduce j_sum1 and tex's pressure!

streety - 8-10-2018 at 19:49

Congrats Melgar!

You probably already know but I think you can turn the alerts off in the profile.

XEvilBestSmilk has just been re-registered and retired by WGTR

XEvilBestSmilk - 9-10-2018 at 07:56

Now the password for this spammer's username will be scrambled with random alpha-numeric characters and animal shapes. Good riddance.

fusso - 9-10-2018 at 10:23

Quote: Originally posted by XEvilBestSmilk

Now the password for this spammer's username will be scrambled with random alpha-numeric characters and animal shapes. Good riddance.

So you think you can fool us by pretending a spambot, PhD?

Sciencemadness Discussion Board - Tired of reporting spam - Powered by XMB 1.9.1.png - 24kB

[Edited on 18/10/09 by fusso]

WGTR - 9-10-2018 at 11:15

No, that was actually me, haha.

That was a particularly obnoxious spammer that I took certain liberties with after I reported it's posts. Once the spammer's posts are removed the account itself is deleted. I simply re-registered the same username to prevent the spammer from reusing it, and then made one post to discourage Bert from deleting it as part of a sweep on zero-post accounts. I do this sometimes when the same name is constantly being registered over and over again. I find this activity to be therapeutic. What kinds of activities on ScienceMadness do you find to be therapeutic?

fusso - 9-10-2018 at 11:35

Quote: Originally posted by WGTR

No, that was actually me, haha.

That was a particularly obnoxious spammer that I took certain liberties with after I reported it's posts. Once the spammer's posts are removed the account itself is deleted. I simply re-registered the same username to prevent the spammer from reusing it, and then made one post to discourage Bert from deleting it as part of a sweep on zero-post accounts. I do this sometimes when the same name is constantly being registered over and over again. I find this activity to be therapeutic. What kinds of activities on ScienceMadness do you find to be therapeutic?

Well I think this question should be discussed in another thread haha

Tsjerk - 9-10-2018 at 20:11

Quote: Originally posted by Magpie

Would it be possible to have the forum software automatically delete:

1. posts in Russian?

cyrillic you mean?

Melgar - 9-10-2018 at 21:40

Ok, my proprietary spam-destroying system "botkilla" has been activated. There was just an insane amount of spam, and even with the threshold turned down pretty low, there wasn't a single false positive. So I'm going to try leaving it on overnight. It runs every three minutes, and so far has gotten all the spam that's been posted in the last few hours. I'll probably tweak the sensitivity tomorrow, if all goes well. In the meantime, if you want to test it, try starting a new thread with the words "mature" or "adult" or "galleries" or Cyrillic characters in the title and see how long it stays up.

I'll make it so that there's additional tests for those cases, but in the meantime, I think most of you will notice a huge drop in the amount of visible spam now. And with the moderators no longer having quite as many spam reports to deal with, we might actually be able to respond to U2U messages, if this actually works.

j_sum1 - 9-10-2018 at 22:36

In the meantime, if you want to test it, try starting a new thread with the words "mature" or "adult" or "galleries" or Cyrillic characters in the title and see how long it stays up.

I did just that. here

It seems to be staying up -- it has been longer than 3 mins and there have been no replies in the thread. I will guess that the reason is that it is noit a new registration. (Does your botkilla test for that?) I might try again later with a dummy registration.

[edit]
Ok. I registered with the name, jsumxgwsukspamtester and posted something with a spammy title. It lasted only a couple of minutes before vanishing.
Account is not deleted or banned but the garbage is gone.

Well done, Melgar.
<clap><clap><clap><clap><clap><clap><clap><clap><clap><clap>

[Edited on 10-10-2018 by j_sum1]

streety - 10-10-2018 at 04:15

Nice to see some automation. From your description and the behavior of your bot I assume you are not re-using any of the program I sent you.

I can also send you all the spam messages my script has collected over the past few months. Hopefully it will help with training your own implementation. I'll wait for the current batch to be cleared.

[edit]Some of the spam posts have 80+ views. I don't think I've ever seen that before. Melgar, will that be your script?

[Edited on 10-10-2018 by streety]

streety - 10-10-2018 at 04:52

The attached file contains a sqlite database file with three tables: topic, member and forum.

This is the definition for the tables:

Code:

from sqlalchemy import Column, Integer, Text, DateTime, ForeignKey
from sqlalchemy.orm import relationship
from sqlalchemy.ext.declarative import declarative_base


Base = declarative_base()



class Topic(Base):
    __tablename__ = 'topic'
    id = Column(Integer, primary_key=True)
    num_posts = Column(Integer, )
    num_views = Column(Integer, )
    first_post = Column(Text, )
    topic_title = Column(Text, )
    topic_id = Column(Integer, index=True, )
    post_date = Column(DateTime)
    delete_date = Column(DateTime, nullable=True, )
    forum_id = Column(Integer, ForeignKey('forum.id'))
    member_id = Column(Integer, ForeignKey('member.id'))

    def __init__(self, num_posts, num_views, first_post,
                topic_title, topic_id, post_date, forum_id, member_id):
        self.num_posts = num_posts
        self.num_views = num_views
        self.first_post = first_post
        self.topic_title = topic_title
        self.topic_id = topic_id
        self.post_date = post_date
        self.forum_id = forum_id
        self.member_id = member_id

    def __repr__(self):
        return '<Topic %r>' % (self.topic_title)


class Member(Base):
    __tablename__ = 'member'
    id = Column(Integer, primary_key=True)
    name = Column(Text, )
    register_date = Column(DateTime)
    deleted_date = Column(DateTime, nullable=True, )
    topics = relationship('Topic', backref='member', lazy='dynamic', )

    def __init__(self, name, register_date):
        self.name = name
        self.register_date = register_date

    def __repr__(self):
        return '<Member %r>' % (self.name)


class Forum(Base):
    __tablename__ = 'forum'
    id = Column(Integer, primary_key=True)
    forum_id = Column(Integer)
    name = Column(Text, )
    topics = relationship('Topic', backref='forum', lazy='dynamic', )

    def __init__(self, forum_id, name):
        self.forum_id = forum_id
        self.name = name

    def __repr__(self):
        return '<Forum %r>' % (self.name)

If a post has been deleted the delete_date field will be set.

Attachment: db.sqlite.tar.gz (4.8MB)
This file has been downloaded 684 times

Melgar - 10-10-2018 at 07:02

I'm posting a more detailed description of what botkilla looks for in Whimsy, but I'll mention here that the script does keep logs of every post it deletes. Here's the one j_sum1 seems to have made:

Code:

title: mature spam poke out
username: jsumxgwsukspamtester
replies: 0
last_poster: jsumxgwsukspamtester
tid: 96794
spam_score: 12
flags:
- spam words in title
- registered since yesterday
thread_text: "Just a quick test of the new botkilla<br />\r\nj_sum1"

Here's the link to a longer description of what it does, for whoever can access Whimsy:

http://www.sciencemadness.org/talk/viewthread.php?tid=96884

I think I'm going to have to have to implement all the rules manually, just because I want to lower the chances of false positives as much as possible.

[Edited on 10/10/18 by Melgar]

JJay - 10-10-2018 at 07:13

Modern antispam software has very little trouble with false positives. Quit making excuses; if you're incompetent, turn in your resignation.

Melgar - 10-10-2018 at 07:19

Yes, I took the script down for like a half hour while I worked on it, and that's what's accumulated during that time period. Look at the times on all of those posts.

fusso - 13-10-2018 at 15:29

0 posts?

Sciencemadness Discussion Board - Rudi Davie chose Royalcolleges Team - Powered .png - 15kB

j_sum1 - 13-10-2018 at 16:31

That's how much post count really means.

Tsjerk - 14-10-2018 at 09:00

@fusso; what do you mean with that post? It is a guy who registered in 2011 and he didn't post anything. So what?

@Melgar; nice work! No more spam for at least a couple of days!

[Edited on 14-10-2018 by Tsjerk]

fusso - 14-10-2018 at 09:19

@fusso; what do you mean with that post? It is a guy who registered in 2011 and he didn't post anything. So what?

I said something wrong?!:O

Tsjerk - 14-10-2018 at 09:54

No, nothing wrong, I just don't understand what you mean with that screenshot.

Ah ok, it is a post but his count is on zero, well, I wouldn't worry too much about it.

[Edited on 14-10-2018 by Tsjerk]

fusso - 14-10-2018 at 10:22

No, nothing wrong, I just don't understand what you mean with that screenshot.

Ah ok, it is a post but his count is on zero, well, I wouldn't worry too much about it.

[Edited on 14-10-2018 by Tsjerk]

My date format is yy/mm/dd, so that bot is registered in 2018, not 2011

The screenshot shows the bot's spam post but its post count is 0, and these 2 observations contradict each other.

[Edited on 18/10/14 by fusso]

Tsjerk - 14-10-2018 at 18:57

Ah, then I understand why I didn't get it

Eidt: All spam is disappearing in minutes!

[Edited on 15-10-2018 by Tsjerk]

Melgar - 16-10-2018 at 08:37

Nice! Glad it's working.

It probably catches a new spam post every 5-10 minutes. There was a bug in my script that would make it stop running if several unlikely things happened at once. It's fixed now though, so that shouldn't happen anymore.

I don't know why that user has no posts, but I recognize that user as a spammer, since I've been periodically checking the logs. And I'm not going to worry about a spammer's post count.

phlogiston - 16-10-2018 at 14:40

Melgar, THANK YOU! It is such a joy to find nearly only non-spam in 'today's post' and browse for interesting contributions again.

fusso - 18-10-2018 at 16:05

@Melgar
Can your program detect spam posts in other threads?

j_sum1 - 18-10-2018 at 20:06

@Melgar
Can your program detect spam posts in other threads?

No.
Report those. Or, preferably, send a mod a u2u. These will need a manual deletion. The volume of "reported post" u2us has gone way down but I still don't read all of them if the board is looking clear of spam. (I presume the other mods are the same.) If you want to attract out attention then a specific message will work better.

Melgar - 18-10-2018 at 20:26

@Melgar
Can your program detect spam posts in other threads?

It could, but doesn't. The main reason is that I haven't seen enough of those types of posts to come up with a good method of identifying them. A spam post would have to stay up long enough for me to write the code to detect it and then test that code on it to make sure that everything works properly. By the time that all could take place, the spam is typically gone, thanks to the quick action of a mod.

I've noticed that spam reports are about 10x as frequent when the script has been temporarily taken down, so I'm assuming that the occasional spam that appends itself to other threads is a pretty minor problem. I could come up with rules to use to eliminate it easily enough, but a few of those posts would need to stay up long enough to make sure those rules work to identify them. I'll leave it up to others to determine whether it's worth a coordinated effort to address.

Mr. Rogers - 20-10-2018 at 20:27

Does the sign-up system use CAPTCHAs? I don't recall. These posts don't look like real people. It's like generic Viagra SPAM.

[Edited on 21-10-2018 by Mr. Rogers]

fusso - 21-10-2018 at 09:01

Quote: Originally posted by Mr. Rogers

Does the sign-up system use CAPTCHAs? I don't recall. These posts don't look like real people. It's like generic Viagra SPAM.

[Edited on 21-10-2018 by Mr. Rogers]

No, no antispam features in registration.

[Edited on 181021 by fusso]

SPAMMERS HAVE EVOLVED

fusso - 22-10-2018 at 09:45

SPAMMERS HAVE EVOLVED TO PRETEND TO BE CHEMISTS :mad:

Melgar - 24-10-2018 at 10:08

SPAMMERS HAVE EVOLVED TO PRETEND TO BE CHEMISTS :mad:

My guess is it's just copying text from some other thread. BotKilla would get that one anyway, because it's linking to an unrecognized domain, and it registered within the last 48 hours.

fusso - 24-10-2018 at 10:32

SPAMMERS HAVE EVOLVED TO PRETEND TO BE CHEMISTS :mad:

My guess is it's just copying text from some other thread. BotKilla would get that one anyway, because it's linking to an unrecognized domain, and it registered within the last 48 hours.

Even if it's only copying, it does seem to know what should it copy

Melgar - 24-10-2018 at 21:33

BotKilla in action.

I sort of gave it an overhaul today, that allowed me to run the spam tests independently of the main script. This makes it a lot easier for me to test spam that isn't caught in the filter and quickly write and test new rules that might do a better job of catching it.

Any spam that does get through the filter, it'd help to move it to Detritus so I can take a look at it later and see how it got through.

streety - 25-10-2018 at 12:16

Are you checking for false positives?

Melgar - 25-10-2018 at 13:33

Yes, so far just that one, and that was because of a glitch in the way links were checked against the whitelist. I PMed him and told him what happened, and included the text of his post in case he wanted to repost it. I don't think he's been back in a few days though. Once I fixed that glitch, it no longer got flagged. Really, the scrutiny is just a lot more on recently-registered users, especially when posting links. His link would have been whitelisted though, had the whitelisting been working correctly at the time. I also raised the threshold for what gets deleted and added a few more conditions that can trigger additional flags. The "linking from unrecognized domain" flag has really been a key predictor of spam, by a long shot. And the "registered today" flag too, just because it focuses on the narrow subset of members who post 99% of the spam.

The algorithm has worked flawlessly, actually, it's just a stupid bug in my code that was responsible for that one false positive.

fusso - 25-10-2018 at 14:50

Maybe botkilla need another feature:
temporarily ban a new account if it has more than a certain # of spam posts deleted, say 5?

Melgar - 25-10-2018 at 18:06

I only have moderation privileges, I think banning someone requires an admin. Not that it matters, since it looks like spam is almost always deleted within a minute after being posted. Really, this is a temporary measure to make this board usable until we can commit to switching forum software, and make sure we're all on the same page as far as how we go about it. Obviously, the sooner we do this, the better.

phlogiston - 31-10-2018 at 15:21

Does BotKilla have a day of or did the spammers find a way to circumvent it?
I just browsed through 5 pages of today's posts and found just 10 'real' threads in them.

Melgar - 31-10-2018 at 18:32

Oops. There was some strange character encoding in one of the spam posts that crashed BotKilla, and I didn't notice what happened until just a little while ago. I think I've covered all the bases now, but the only way to know for sure is to just let it run and keep checking it.

Really though, this is turning into quite a lot of responsibility. It's nothing I can't handle, but it would certainly help if there was compensation that came with it. I guess I should probably start a new thread on the subject to see what people think.

fusso - 1-11-2018 at 02:05

Really though, this is turning into quite a lot of responsibility. It's nothing I can't handle, but it would certainly help if there was compensation that came with it. I guess I should probably start a new thread on the subject to see what people think.

Maybe put the thread in whimsy?

fusso - 1-11-2018 at 11:39

Hidden spam in signature:
https://www.sciencemadness.org/whisper/viewthread.php?tid=65...

unionised - 4-11-2018 at 02:41

What the hell is it with mangosteens?
Anyway, I thought I'd say thanks to whoever else was involved in the episode about 15 minutes ago where a whole bunch of spam was quickly detected + wiped.
Very satisfying.

j_sum1 - 4-11-2018 at 02:55

Just you and Melgar's botkilla this thme unionised.

I got no idea what a mangosteen is either and I am no googling it.

unionised - 4-11-2018 at 04:57

Nice to know I achieved something today.
A mangosteen is a fruit, BTW, Nice enough. Nothing special except it's a bit of a novelty for most people in the
UK or US

fusso - 4-11-2018 at 06:12

Quote: Originally posted by unionised

Nice to know I achieved something today.
A mangosteen is a fruit, BTW, Nice enough. Nothing special except it's a bit of a novelty for most people in the
UK or US

Mangosteen is also a...typhoon aka hurricane/cyclone

Tsjerk - 4-11-2018 at 08:04

Very nice work done Melgar! could I have a copy of your code? Out of curiosity

Melgar - 4-11-2018 at 08:18

Quote: Originally posted by unionised

What the hell is it with mangosteens?
Anyway, I thought I'd say thanks to whoever else was involved in the episode about 15 minutes ago where a whole bunch of spam was quickly detected + wiped.
Very satisfying.

Oh right. If my U2U message count shoots up due to spam reports, that's my cue to check on the BotKilla script. Indeed, it had malfunctioned and needed to be restarted. Since I let this thing loose with a lot of untested code initially, I've been operating under the principle that it's better to have it stop when it malfunctions. That way, I can see what caused the problem, and there isn't a malfunctioning bot on the loose, deleting posts. But like the last three times that's happened, it's been some sort of temporary network error, so next time that happens I'll set it to ignore that specific error and just keep trying again until it works.

When it finally does get running again, it keeps track of how many spam posts it's deleted per hour, and I admit that seeing spam text scroll past faster than I can read it, and watching that rate temporarily shoot up into the tens of thousands is quite satisfying.

Melgar - 4-11-2018 at 08:41

Very nice work done Melgar! could I have a copy of your code? Out of curiosity

Ok, here's the main script :

https://github.com/toldani/sm-transition/blob/master/botkill...

And here's the domain whitelist that links are checked against:

https://github.com/toldani/sm-transition/blob/master/sm-link...

No single flag is enough to trigger a deletion, but most spam is deleted when a user's first post includes a link to a domain not on the whitelist. The whitelist was generated by scanning 14 years of forum history and storing all the domains that were linked to at least ten times.

unionised - 4-11-2018 at 14:44

I'm sure teh question was asked earlier but... can you get the code to spot posts in cyrillic?

Melgar - 4-11-2018 at 15:51

Quote: Originally posted by unionised

I'm sure teh question was asked earlier but... can you get the code to spot posts in cyrillic?

Yeah, it already does. Also, Chinese, Tagalog, Arabic, and Thai. Although I've really only seen much Cyrillic and Chinese.

Right now, it only gets new threads and not spam that's been tacked onto the end of existing threads, but that's mostly because that spam is less common and it usually gets deleted before I have a chance to figure out how to detect it.

Texium - 13-11-2018 at 10:14

In case anyone was wondering, I reported the Melgar impersonator as Spam in order to swiftly destroy it. So that's where it went.

Melgar - 18-11-2018 at 18:07

Oh, ok. I was kinda wondering what happened with that.

It's nice to see so many new threads being started that aren't spam.

Also, I've configured BotKilla to wait five minutes after it crashes, then start running again. The main thing crashing it now seems to actually be if a thread gets deleted between when BotKilla determines a thread to be spam, and when it tries to navigate to that thread to delete it. So really it was just a 404 error that only became a problem because I hadn't set up error handling in the "delete thread" function. But since it seems to cause way more problems when it's down than when it's up and malfunctioning slightly, it should now recover from any errors that it throws, and go back to hunting spam.

Mr. Rogers - 19-11-2018 at 08:45

I haven't seen any spam in two days. Something is obviously working.

12thealchemist - 19-11-2018 at 11:55

I'm afraid you've just jinxed it. Two spam posts this evening, but different to the usual ones - paper writing and increased business

Melgar - 19-11-2018 at 13:45

I lowered the refresh time from 60 seconds to 30 seconds, meaning that spam that's slated for automatic deletion will only be visible half as long now. It's not actually that big of a deal, but if you guys don't mind, could you just wait about a minute to see if spam gets deleted automatically before reporting it? Most of the spam that gets reported actually does end up getting auto-deleted a few seconds later, actually.

The only spam I've seen getting through regularly are these super vague posts with no links and terrible English. Since those are the epitome of pointlessness though, I'm not worried about them, and have just been deleting those manually.

fusso - 19-11-2018 at 13:45

It's nice to see so many new threads being started that aren't spam.

Also, I've configured BotKilla to wait five minutes after it crashes, then start running again.

Why not config it to restart sooner, preferably instantly?

Melgar - 19-11-2018 at 13:53