Sciencemadness Discussion Board
Not logged in [Login - Register]
Go To Bottom

Printable Version  
 Pages:  1  2
Author: Subject: Lengthy, time-consuming overhaul of forum software
JJay
International Hazard
*****




Posts: 2901
Registered: 15-10-2015
Location: Western Hemisphere
Member Is Offline

Mood: planning a semi-novel acetonitrile synthesis

[*] posted on 20-2-2018 at 14:29


Amazon's cloud services are the best. Google's come close, and Google can do some things that Amazon can't (the Cloud Machine Learning Engine and Cloud Spanner come to mind). Also, Google gives everyone a free cloud node. Yet most of Amazon's services are more advanced than Google's. Amazon also allows doing things like building a network appliance using a standard file format in VirtualBox and uploading it directly to create a machine instance (Google requires uploading a disk image).



This is my YouTube channel: Extreme Red Cabbage. I don't have much posted, but I try to do nice writeups once in a while.
View user's profile View All Posts By User
Melgar
International Hazard
*****




Posts: 1747
Registered: 23-2-2010
Location: NYC
Member Is Offline

Mood: Aromatic

[*] posted on 20-2-2018 at 22:47


Personally, I made most extensive use of Amazon's S3, but generally like the fact that when you start a project, you can add services as you need them rather than try to estimate what you'll eventually use.

The main annoying thing about AWS is that they use a lot of their own terminology that you have to learn, and they tend to solve problems by adding complexity.

I noticed AWS has machine learning now too. It must have been added just recently.




The first step in the process of learning something is admitting that you don't know it already.
View user's profile View All Posts By User
JJay
International Hazard
*****




Posts: 2901
Registered: 15-10-2015
Location: Western Hemisphere
Member Is Offline

Mood: planning a semi-novel acetonitrile synthesis

[*] posted on 21-2-2018 at 01:10


AWS has had it for a lot longer actually, but comparing their machine learning capabilities to Google's is like comparing a BB gun to a tank.



This is my YouTube channel: Extreme Red Cabbage. I don't have much posted, but I try to do nice writeups once in a while.
View user's profile View All Posts By User
froot
National Hazard
****




Posts: 338
Registered: 23-10-2003
Location: South Africa
Member Is Offline

Mood: refluxed

[*] posted on 21-2-2018 at 11:04


Hosting is not really as big an issue as software conversion. You can move hosts before or after converting, or not converting, having worked with phpbb as an admin let me suggest another option.

Install the latest Phpbb and sql database alongside this forum and make front end access to that.

Find the most appropriate 'skin' and set up the discussion layout as it is here. One can also try copy, edit and re-use layout templates from this forum to make it look almost identical.

In each topic section add another forum/topic at the bottom calling it 'archive' or something with a link to the same forum/topic on this board which has been closed from posting and for reference only.

If a user wants to contribute to a topic here in the archive you can possibly write a script that automatically opens a new topic with a link to where reply was meant to go in the new forum and the discussion can continue relatively seamlessly.

Once you've set everything up point the domain to the new phpbb forum and make the old one inaccessible without registration.

The advantages are:

All links in the old database remain valid.

No need to double up on temporary disk space while doing the conversion.

Much quicker.

Nothing lost.

Negligible downtime.

This board's vulnerabilities can be hidden behind the new forum's security.

Some of this board's functionality becomes dispensible.

The disadvantages:

Users will have to re-register on the new forum. Once they've registered they will be able to access their old accounts on this board. Most people will carry their old usernames with them but there will always be the anomalies.

I think it's a no-brainer unless future hosting will render XMB absolutely dead.







We salute the improvement of the human genome by honoring those who remove themselves from it.
Of necessity, this honor is generally bestowed posthumously. - www.darwinawards.com
View user's profile View All Posts By User
Melgar
International Hazard
*****




Posts: 1747
Registered: 23-2-2010
Location: NYC
Member Is Offline

Mood: Aromatic

[*] posted on 21-2-2018 at 14:44


Yeah, I've adminned phpBB too. The difference seems to be that I'm also a programmer. Since all of this software is open source, and because the data exists entirely in SQL tables that can be easily queried and manipulated, it's possible to migrate from XMB to phpBB without any loss of data. The main issue is that Polverone controls access to the server, and is busy most of the time. However, he's distributed a virtual machine containing SM archives, with passwords reset and private messages erased. That's the database that I have. The goal here is to refactor the database to phpBB's format, then just make sure that can be done with a script. Then Polverone could run the scripts with the backup database on a new server, and it should be up and running.

I've been busy with other things the last 2-3 days, but it's in the works.




The first step in the process of learning something is admitting that you don't know it already.
View user's profile View All Posts By User
madcedar
Hazard to Others
***




Posts: 107
Registered: 10-9-2009
Member Is Offline

Mood: No Mood

[*] posted on 21-2-2018 at 16:26


Melgar, you're a star mate. If you are successful with this transfer does that mean we'll be able to view the forum with ease on a mobile device?

Cheers
View user's profile View All Posts By User
Melgar
International Hazard
*****




Posts: 1747
Registered: 23-2-2010
Location: NYC
Member Is Offline

Mood: Aromatic

[*] posted on 8-3-2018 at 01:29


Well, I hope I finish it. If it works, then it'll be able to do whatever the most recent version of phpBB can do. (I haven't used it in a while, and I honestly don't know)



The first step in the process of learning something is admitting that you don't know it already.
View user's profile View All Posts By User
Melgar
International Hazard
*****




Posts: 1747
Registered: 23-2-2010
Location: NYC
Member Is Offline

Mood: Aromatic

[*] posted on 9-3-2018 at 19:22


I concluded that the best way to migrate the data is by having mySQL output a tab-delineated csv-type file, then import that data into postgres using COPY. It's sort of annoying how XMB will have a SET-type column that contains two values, "on" and "off", but heaven forbid they just make the column a boolean. Then they make the default an empty string, which is totally different from NULL, and not part of the set anyway. I should probably just make all those weird columns strings, then write functions to convert them to their phpBB counterparts. I'll have to do that anyway for a lot of data fields, and it'd be easy to just make that part of the process. If possible, I'd like to have php handlers that are either at the same url as the XMB versions, or at least redirect to the phpBB urls. Thread and post IDs should be easy enough to keep unchanged.

I've been running into issues of money, actually. My internet has been predominantly through tethering to my phone, and the data charges have added up. I just got $200 via paypal from my aspiring alchemist friend, which helps. And I'm having to spend a lot of time compiling evidence against my former employer just to collect unemployment. But once that happens, I should get months of unemployment all at once. It's been a rough couple of months though.

If anyone wants to help, it would be awesome if someone could purchase a Dash license for me. It's an offline documentation browser, which would allow me to download all the programming documentation I'd need via public wifi, then use it for reference when I'm paying for data via my phone. So far, I've picked up the tab for AWS fees, but fortunately they haven't been much.

Dash licenses can be used on multiple devices too, so if you have a license or want one for yourself, you can share it. I used to have a license, but then they released a new version and required you to purchase an upgrade. I missed the deadline where I could upgrade at a lower price though.




The first step in the process of learning something is admitting that you don't know it already.
View user's profile View All Posts By User
Melgar
International Hazard
*****




Posts: 1747
Registered: 23-2-2010
Location: NYC
Member Is Offline

Mood: Aromatic

[*] posted on 7-4-2018 at 20:39


Good news! I figured out a straightforward way to copy all the data over. First I had to completely rewrite the schema, to make sure data types were compatible. But I finally got the "forums" table, the "members" table, and the "posts" table to import successfully into Postgres. The data had to be exported via a CSV though, otherwise the encoding got messed up, and none of the fixes seemed easy.

To output a table in mySQL as CSV, this was how I did it:

Code:
SELECT * INTO OUTFILE '/tmp/sm_threads.csv' FIELDS TERMINATED BY ',' OPTIONALLY ENCLOSED BY '"' LINES TERMINATED BY '\n' FROM xmb_threads;


Then I just uploaded all the CSVs to the host machine via scp to import them. A typical Postgres command to import a CSV is:

Code:
\copy xmb_posts FROM '/home/ubuntu/sm_posts.csv' WITH (FORMAT CSV, ESCAPE '\')


You can also use the standard SQL COPY command, but there are issues with permissions that are annoying.

I'm also realizing that most of the tables won't even need to be transferred over, like "ban" tables, and XMB settings tables. Easier just to figure that out via the web interface.

The important part is that I've figured out how to extract data from really old software that I'm unfamiliar with, and import it all into a more modern database system.

Once the imported tables are in the same database as the (empty) phpBB tables, I can start writing functions to format the XMB data for phpBB. But I need to play around with phpBB a bit first.

Attachments are stored really inefficiently, so I plan to write a script to store them in AWS S3, which charges about 1/3 the price. It also lets us get away with a much smaller primary volume.

My plan was to use nginx instead of Apache, mainly because you can make it as simple or complex as you want, whereas Apache is always pretty complex. Also, I'm more familiar with it, and the learning curve seems lower to me. They're practically neck and neck as far as popularity, but I've always had an easier time getting necessarily-bizarre configurations working with nginx.

Presumably Polverone can understand the process well enough to set it up once I do all the legwork.

Would anyone care if I made it HTTPS exclusively, and made "whisper" redirect to "talk"? Also, what's up with our wiki? Seems like half the time I see it, it's full of ads for old sci-fi shows. Easiest would be to host it all one one server, I'd imagine.




The first step in the process of learning something is admitting that you don't know it already.
View user's profile View All Posts By User
DavidJR
Hazard to Others
***




Posts: 130
Registered: 1-1-2018
Location: Scotland
Member Is Offline

Mood: Anxious

[*] posted on 7-4-2018 at 21:02


Quote: Originally posted by Melgar  

Would anyone care if I made it HTTPS exclusively, and made "whisper" redirect to "talk"?

There's absolutely no good reason to have the site accessible via http if https is already set up, and even less reason to have separate sub-paths /talk and /whisper ?! So that sounds like a good idea. Just be sure to set up redirects properly so no links are broken.

Quote: Originally posted by Melgar  

Also, what's up with our wiki? Seems like half the time I see it, it's full of ads for old sci-fi shows. Easiest would be to host it all one one server, I'd imagine.

Do you mean the old wikia one? Personally I think that should be deleted since the wiki moved 'home'
View user's profile View All Posts By User
Melgar
International Hazard
*****




Posts: 1747
Registered: 23-2-2010
Location: NYC
Member Is Offline

Mood: Aromatic

[*] posted on 8-4-2018 at 11:58


Okay, basically what I'm not sure of is how much of this stuff Polverone can figure out on his own. I'd appreciate it if we could just spend some time on the phone when we're ready to switch. Just because I don't know him well enough to know what he's comfortable with. On the one hand, he's certainly someone I look up to, but on the other hand, it would really suck if I overestimated him Also, I accidentally deleted the attachment data from the virtual machine, but Polverone should be able to write a script easy enough to convert them into files. Then it'd just be a matter of using some regex magic to fix all the links to attachments. I believe attachments are in base64 format, which can be easily decoded via the command line like so:

printf "SGV5LCB5b3UgY2FuIHVzZSBhIHRlcm1pbmFsISBXYW50IHRvIGhlbHAgdXBncmFkZSBzY2llbmNlbWFkbmVzcy5vcmc/IDpE" | base64 -d

The "-d" is capitalized on Mac/BSD.

If it's binary, it can just be piped to an output file, and a script could handle them easily enough. My real point is that I don't want to reset the virtual machine just to get the attachments table, since it wouldn't be complete anyway, and parsing it into files would be simple. I actually have the metadata for the files, but erased to data column for attachments, which was 90% of the database size.

I'm also looking into linking an ftp client to the users database, and then figuring out some criteria for posting there. (100 posts over 3 months, maybe?) Files would be available automatically via the s3 bucket. I set up s3fs and confirmed that it works, I just haven't figured out the best way to make FTP work, which is a low priority anyway.

Next step is configuring phpBB before setting it up, to make sure that paths are as close to they are now as possible. Fortunately, phpBB is open-source, and changing something little like that should be simple.




The first step in the process of learning something is admitting that you don't know it already.
View user's profile View All Posts By User
Tsjerk
International Hazard
*****




Posts: 715
Registered: 20-4-2005
Location: Netherlands
Member Is Offline

Mood: Mood

[*] posted on 9-4-2018 at 02:01


@Melgar; I didn't read this thread before, but you are a hero. Don't forget your cape!

Which version of phpBB are you using? 3.2.2?

Edit; could you let me know when you have the board running somewhere? Also when it is not for public yet? I want to try some attacks. I don't think I can do any damage, but I like to try.


[Edited on 9-4-2018 by Tsjerk]
View user's profile View All Posts By User
Melgar
International Hazard
*****




Posts: 1747
Registered: 23-2-2010
Location: NYC
Member Is Offline

Mood: Aromatic

[*] posted on 19-4-2018 at 11:01


I got a very rudimentary mockup of this site in phpBB. Don't worry about the theme though, that can easily be customized:

http://ec2-184-72-122-13.compute-1.amazonaws.com/talk/index....

I need a bunch of sample posts and such in order to reverse engineer our older data, so have at it. Note that permissions have not been set properly, but there's nothing there to read yet, so that's not actually a problem.

Feedback is appreciated. If you're not an obvious troll, let me know and I'll give you mod/admin privileges. (Do it here though, so I can confirm who you are)




The first step in the process of learning something is admitting that you don't know it already.
View user's profile View All Posts By User
Corrosive Joeseph
National Hazard
****




Posts: 429
Registered: 17-5-2015
Location: Europe
Member Is Offline

Mood: Cyclic

[*] posted on 19-4-2018 at 11:14


Looks exactly like an engine management forum I know.
Maybe this might give peeps an example preview

http://www.hrtuning.com/forum/


/CJ




MSDS Addict and OTC Slut
View user's profile View All Posts By User
Melgar
International Hazard
*****




Posts: 1747
Registered: 23-2-2010
Location: NYC
Member Is Offline

Mood: Aromatic

[*] posted on 19-4-2018 at 15:26


Yep, it says "phpBB" right at the bottom, and it clearly uses the default theme. It's really easy to have a new theme for a board though, and there are plenty of public ones out there to draw from. The point was that I need people to post and register and all that on this board, in order to make it easier to reverse-engineer the XMB format.

My point was to try to make everything seem like the current boards, but with a different theme (that can easily be changed).




The first step in the process of learning something is admitting that you don't know it already.
View user's profile View All Posts By User
Tsjerk
International Hazard
*****




Posts: 715
Registered: 20-4-2005
Location: Netherlands
Member Is Offline

Mood: Mood

[*] posted on 19-4-2018 at 15:38


Hello Melgar, nice going! I just registered under "Tsjerk". I will make some posts
View user's profile View All Posts By User
Melgar
International Hazard
*****




Posts: 1747
Registered: 23-2-2010
Location: NYC
Member Is Offline

Mood: Aromatic

[*] posted on 19-4-2018 at 17:32


Thanks! The database formats are different enough that I've been having to keep an eye on them to see how activity in them changes table data.

I appreciate ANY feedback on the way the site is set up and looks, EXCEPT as it relates to the theme. No sense in building a custom them if nobody ends up being interested in using it. So yeah, please have all sorts of discussions there, including trolling and being mean and posting "foolproof" meth syntheses you thought up after binge-watching Breaking Bad. :P

The content doesn't matter. I just need to be able to compare how the two database systems store data




The first step in the process of learning something is admitting that you don't know it already.
View user's profile View All Posts By User
 Pages:  1  2

  Go To Top