| Pages:
1
2
3
4 |
JJay
International Hazard
Posts: 3440
Registered: 15-10-2015
Member Is Offline
|
|
Amazon's cloud services are the best. Google's come close, and Google can do some things that Amazon can't (the Cloud Machine Learning Engine and
Cloud Spanner come to mind). Also, Google gives everyone a free cloud node. Yet most of Amazon's services are more advanced than Google's. Amazon also
allows doing things like building a network appliance using a standard file format in VirtualBox and uploading it directly to create a machine
instance (Google requires uploading a disk image).
|
|
|
Melgar
Anti-Spam Agent
Posts: 2004
Registered: 23-2-2010
Location: Connecticut
Member Is Offline
Mood: Estrified
|
|
Personally, I made most extensive use of Amazon's S3, but generally like the fact that when you start a project, you can add services as you need them
rather than try to estimate what you'll eventually use.
The main annoying thing about AWS is that they use a lot of their own terminology that you have to learn, and they tend to solve problems by adding
complexity.
I noticed AWS has machine learning now too. It must have been added just recently.
The first step in the process of learning something is admitting that you don't know it already.
I'm givin' the spam shields max power at full warp, but they just dinna have the power! We're gonna have to evacuate to new forum software!
|
|
|
JJay
International Hazard
Posts: 3440
Registered: 15-10-2015
Member Is Offline
|
|
AWS has had it for a lot longer actually, but comparing their machine learning capabilities to Google's is like comparing a BB gun to a tank.
|
|
|
froot
Hazard to Others
Posts: 347
Registered: 23-10-2003
Location: South Africa
Member Is Offline
Mood: refluxed
|
|
Hosting is not really as big an issue as software conversion. You can move hosts before or after converting, or not converting, having worked with
phpbb as an admin let me suggest another option.
Install the latest Phpbb and sql database alongside this forum and make front end access to that.
Find the most appropriate 'skin' and set up the discussion layout as it is here. One can also try copy, edit and re-use layout templates from this
forum to make it look almost identical.
In each topic section add another forum/topic at the bottom calling it 'archive' or something with a link to the same forum/topic on this board which
has been closed from posting and for reference only.
If a user wants to contribute to a topic here in the archive you can possibly write a script that automatically opens a new topic with a link to where
reply was meant to go in the new forum and the discussion can continue relatively seamlessly.
Once you've set everything up point the domain to the new phpbb forum and make the old one inaccessible without registration.
The advantages are:
All links in the old database remain valid.
No need to double up on temporary disk space while doing the conversion.
Much quicker.
Nothing lost.
Negligible downtime.
This board's vulnerabilities can be hidden behind the new forum's security.
Some of this board's functionality becomes dispensible.
The disadvantages:
Users will have to re-register on the new forum. Once they've registered they will be able to access their old accounts on this board. Most people
will carry their old usernames with them but there will always be the anomalies.
I think it's a no-brainer unless future hosting will render XMB absolutely dead.
We salute the improvement of the human genome by honoring those who remove themselves from it.
Of necessity, this honor is generally bestowed posthumously. - www.darwinawards.com |
|
|
Melgar
Anti-Spam Agent
Posts: 2004
Registered: 23-2-2010
Location: Connecticut
Member Is Offline
Mood: Estrified
|
|
Yeah, I've adminned phpBB too. The difference seems to be that I'm also a programmer. Since all of this software is open source, and because the data
exists entirely in SQL tables that can be easily queried and manipulated, it's possible to migrate from XMB to phpBB without any loss of data. The
main issue is that Polverone controls access to the server, and is busy most of the time. However, he's distributed a virtual machine containing SM
archives, with passwords reset and private messages erased. That's the database that I have. The goal here is to refactor the database to phpBB's
format, then just make sure that can be done with a script. Then Polverone could run the scripts with the backup database on a new server, and it
should be up and running.
I've been busy with other things the last 2-3 days, but it's in the works.
The first step in the process of learning something is admitting that you don't know it already.
I'm givin' the spam shields max power at full warp, but they just dinna have the power! We're gonna have to evacuate to new forum software!
|
|
|
madcedar
Hazard to Others
Posts: 116
Registered: 10-9-2009
Member Is Offline
Mood: No Mood
|
|
Melgar, you're a star mate. If you are successful with this transfer does that mean we'll be able to view the forum with ease on a mobile device?
Cheers
|
|
|
Melgar
Anti-Spam Agent
Posts: 2004
Registered: 23-2-2010
Location: Connecticut
Member Is Offline
Mood: Estrified
|
|
Well, I hope I finish it. If it works, then it'll be able to do whatever the most recent version of phpBB can do. (I haven't used it in a while, and
I honestly don't know)
The first step in the process of learning something is admitting that you don't know it already.
I'm givin' the spam shields max power at full warp, but they just dinna have the power! We're gonna have to evacuate to new forum software!
|
|
|
Melgar
Anti-Spam Agent
Posts: 2004
Registered: 23-2-2010
Location: Connecticut
Member Is Offline
Mood: Estrified
|
|
I concluded that the best way to migrate the data is by having mySQL output a tab-delineated csv-type file, then import that data into postgres using
COPY. It's sort of annoying how XMB will have a SET-type column that contains two values, "on" and "off", but heaven forbid they just make the column
a boolean. Then they make the default an empty string, which is totally different from NULL, and not part of the set anyway. I should probably just
make all those weird columns strings, then write functions to convert them to their phpBB counterparts. I'll have to do that anyway for a lot of data
fields, and it'd be easy to just make that part of the process. If possible, I'd like to have php handlers that are either at the same url as the XMB
versions, or at least redirect to the phpBB urls. Thread and post IDs should be easy enough to keep unchanged.
I've been running into issues of money, actually. My internet has been predominantly through tethering to my phone, and the data charges have added
up. I just got $200 via paypal from my aspiring alchemist friend, which helps. And I'm having to spend a lot of time compiling evidence against my
former employer just to collect unemployment. But once that happens, I should get months of unemployment all at once. It's been a rough couple of
months though.
If anyone wants to help, it would be awesome if someone could purchase a Dash license for me. It's an offline documentation browser, which would allow me to download all the programming documentation I'd need via
public wifi, then use it for reference when I'm paying for data via my phone. So far, I've picked up the tab for AWS fees, but fortunately they
haven't been much.
Dash licenses can be used on multiple devices too, so if you have a license or want one for yourself, you can share it. I used to have a license, but
then they released a new version and required you to purchase an upgrade. I missed the deadline where I could upgrade at a lower price though.
The first step in the process of learning something is admitting that you don't know it already.
I'm givin' the spam shields max power at full warp, but they just dinna have the power! We're gonna have to evacuate to new forum software!
|
|
|
Melgar
Anti-Spam Agent
Posts: 2004
Registered: 23-2-2010
Location: Connecticut
Member Is Offline
Mood: Estrified
|
|
Good news! I figured out a straightforward way to copy all the data over. First I had to completely rewrite the schema, to make sure data types were
compatible. But I finally got the "forums" table, the "members" table, and the "posts" table to import successfully into Postgres. The data had to
be exported via a CSV though, otherwise the encoding got messed up, and none of the fixes seemed easy.
To output a table in mySQL as CSV, this was how I did it:
| Code: | SELECT * INTO OUTFILE '/tmp/sm_threads.csv' FIELDS TERMINATED BY ',' OPTIONALLY ENCLOSED BY '"' LINES TERMINATED BY '\n' FROM xmb_threads; |
Then I just uploaded all the CSVs to the host machine via scp to import them. A typical Postgres command to import a CSV is:
| Code: | \copy xmb_posts FROM '/home/ubuntu/sm_posts.csv' WITH (FORMAT CSV, ESCAPE '\') |
You can also use the standard SQL COPY command, but there are issues with permissions that are annoying.
I'm also realizing that most of the tables won't even need to be transferred over, like "ban" tables, and XMB settings tables. Easier just to figure
that out via the web interface.
The important part is that I've figured out how to extract data from really old software that I'm unfamiliar with, and import it all into a more
modern database system.
Once the imported tables are in the same database as the (empty) phpBB tables, I can start writing functions to format the XMB data for phpBB. But I
need to play around with phpBB a bit first.
Attachments are stored really inefficiently, so I plan to write a script to store them in AWS S3, which charges about 1/3 the price. It also lets us
get away with a much smaller primary volume.
My plan was to use nginx instead of Apache, mainly because you can make it as simple or complex as you want, whereas Apache is always pretty complex.
Also, I'm more familiar with it, and the learning curve seems lower to me. They're practically neck and neck as far as popularity, but I've always
had an easier time getting necessarily-bizarre configurations working with nginx.
Presumably Polverone can understand the process well enough to set it up once I do all the legwork.
Would anyone care if I made it HTTPS exclusively, and made "whisper" redirect to "talk"? Also, what's up with our wiki? Seems like half the time I
see it, it's full of ads for old sci-fi shows. Easiest would be to host it all one one server, I'd imagine.
The first step in the process of learning something is admitting that you don't know it already.
I'm givin' the spam shields max power at full warp, but they just dinna have the power! We're gonna have to evacuate to new forum software!
|
|
|
DavidJR
National Hazard
Posts: 908
Registered: 1-1-2018
Location: Scotland
Member Is Offline
Mood: Tired
|
|
There's absolutely no good reason to have the site accessible via http if https is already set up, and even less reason to have separate sub-paths
/talk and /whisper ?! So that sounds like a good idea. Just be sure to set up redirects properly so no links are broken.
Quote: Originally posted by Melgar  |
Also, what's up with our wiki? Seems like half the time I see it, it's full of ads for old sci-fi shows. Easiest would be to host it all one one
server, I'd imagine.
|
Do you mean the old wikia one? Personally I think that should be deleted since the wiki moved 'home'
|
|
|
Melgar
Anti-Spam Agent
Posts: 2004
Registered: 23-2-2010
Location: Connecticut
Member Is Offline
Mood: Estrified
|
|
Okay, basically what I'm not sure of is how much of this stuff Polverone can figure out on his own. I'd appreciate it if we could just spend some
time on the phone when we're ready to switch. Just because I don't know him well enough to know what he's comfortable with. On the one hand, he's
certainly someone I look up to, but on the other hand, it would really suck if I overestimated him Also, I accidentally deleted the attachment data
from the virtual machine, but Polverone should be able to write a script easy enough to convert them into files. Then it'd just be a matter of using
some regex magic to fix all the links to attachments. I believe attachments are in base64 format, which can be easily decoded via the command line
like so:
printf "SGV5LCB5b3UgY2FuIHVzZSBhIHRlcm1pbmFsISBXYW50IHRvIGhlbHAgdXBncmFkZSBzY2llbmNlbWFkbmVzcy5vcmc/IDpE" | base64 -d
The "-d" is capitalized on Mac/BSD.
If it's binary, it can just be piped to an output file, and a script could handle them easily enough. My real point is that I don't want to reset the
virtual machine just to get the attachments table, since it wouldn't be complete anyway, and parsing it into files would be simple. I actually have
the metadata for the files, but erased to data column for attachments, which was 90% of the database size.
I'm also looking into linking an ftp client to the users database, and then figuring out some criteria for posting there. (100 posts over 3 months,
maybe?) Files would be available automatically via the s3 bucket. I set up s3fs and confirmed that it works, I just haven't figured out the best way
to make FTP work, which is a low priority anyway.
Next step is configuring phpBB before setting it up, to make sure that paths are as close to they are now as possible. Fortunately, phpBB is
open-source, and changing something little like that should be simple.
The first step in the process of learning something is admitting that you don't know it already.
I'm givin' the spam shields max power at full warp, but they just dinna have the power! We're gonna have to evacuate to new forum software!
|
|
|
Tsjerk
International Hazard
Posts: 3022
Registered: 20-4-2005
Location: Netherlands
Member Is Offline
Mood: Mood
|
|
@Melgar; I didn't read this thread before, but you are a hero. Don't forget your cape!
Which version of phpBB are you using? 3.2.2?
Edit; could you let me know when you have the board running somewhere? Also when it is not for public yet? I want to try some attacks. I don't think I
can do any damage, but I like to try.
[Edited on 9-4-2018 by Tsjerk]
|
|
|
Melgar
Anti-Spam Agent
Posts: 2004
Registered: 23-2-2010
Location: Connecticut
Member Is Offline
Mood: Estrified
|
|
I got a very rudimentary mockup of this site in phpBB. Don't worry about the theme though, that can easily be customized:
http://ec2-184-72-122-13.compute-1.amazonaws.com/talk/index....
I need a bunch of sample posts and such in order to reverse engineer our older data, so have at it. Note that permissions have not been set properly,
but there's nothing there to read yet, so that's not actually a problem.
Feedback is appreciated. If you're not an obvious troll, let me know and I'll give you mod/admin privileges. (Do it here though, so I can confirm
who you are)
The first step in the process of learning something is admitting that you don't know it already.
I'm givin' the spam shields max power at full warp, but they just dinna have the power! We're gonna have to evacuate to new forum software!
|
|
|
Corrosive Joeseph
National Hazard
Posts: 915
Registered: 17-5-2015
Location: The Other Place
Member Is Offline
Mood: Cyclic
|
|
Looks exactly like an engine management forum I know.
Maybe this might give peeps an example preview
http://www.hrtuning.com/forum/
/CJ
Being well adjusted to a sick society is no measure of one's mental health
|
|
|
Melgar
Anti-Spam Agent
Posts: 2004
Registered: 23-2-2010
Location: Connecticut
Member Is Offline
Mood: Estrified
|
|
Yep, it says "phpBB" right at the bottom, and it clearly uses the default theme. It's really easy to have a new theme for a board though, and there
are plenty of public ones out there to draw from. The point was that I need people to post and register and all that on this board, in order to make
it easier to reverse-engineer the XMB format.
My point was to try to make everything seem like the current boards, but with a different theme (that can easily be changed).
The first step in the process of learning something is admitting that you don't know it already.
I'm givin' the spam shields max power at full warp, but they just dinna have the power! We're gonna have to evacuate to new forum software!
|
|
|
Tsjerk
International Hazard
Posts: 3022
Registered: 20-4-2005
Location: Netherlands
Member Is Offline
Mood: Mood
|
|
Hello Melgar, nice going! I just registered under "Tsjerk". I will make some posts
|
|
|
Melgar
Anti-Spam Agent
Posts: 2004
Registered: 23-2-2010
Location: Connecticut
Member Is Offline
Mood: Estrified
|
|
Thanks! The database formats are different enough that I've been having to keep an eye on them to see how activity in them changes table data.
I appreciate ANY feedback on the way the site is set up and looks, EXCEPT as it relates to the theme. No sense in building a custom them if nobody
ends up being interested in using it. So yeah, please have all sorts of discussions there, including trolling and being mean and posting "foolproof"
meth syntheses you thought up after binge-watching Breaking Bad. 
The content doesn't matter. I just need to be able to compare how the two database systems store data
The first step in the process of learning something is admitting that you don't know it already.
I'm givin' the spam shields max power at full warp, but they just dinna have the power! We're gonna have to evacuate to new forum software!
|
|
|
madcedar
Hazard to Others
Posts: 116
Registered: 10-9-2009
Member Is Offline
Mood: No Mood
|
|
One feature that I love about the current forum site is the "Today's Posts" button at the top. Is it possible to add this functionality to the new
site or is that part of a theme?
Edit: It's okay, I've just noticed that the new software has an Active Topics button that can be filtered to one day.
[Edited on 20-4-2018 by madcedar]
|
|
|
Melgar
Anti-Spam Agent
Posts: 2004
Registered: 23-2-2010
Location: Connecticut
Member Is Offline
Mood: Estrified
|
|
Anyone interested in commissioning an artist to redo the theme for this site? We'd try to keep a similar look and feel, though perhaps also with the
look and feel of a well-utilized workshop somehow. Everyone would be able to contribute ideas, if they wanted to.
This wouldn't happen until the site is fully transferred, but it would be nice to know if that's something that's in the works.
Also, please, everyone, go there and register and try and use as many features as you can. I need data to reverse engineer this shit.
The first step in the process of learning something is admitting that you don't know it already.
I'm givin' the spam shields max power at full warp, but they just dinna have the power! We're gonna have to evacuate to new forum software!
|
|
|
chemplayer...
Hazard to Others
Posts: 191
Registered: 25-4-2016
Location: Away from the secret island
Member Is Offline
Mood: No Mood
|
|
Had a play and it feels good. Differences I can see are:
- Less intuitive to add hyperlinks, images, etc. The buttons at the top of the full post editing screen add [] square bracket code but it's hard to
know how to use these (e.g. lists are not easy to do without experimentation), and no visualisation on-screen of the result unless you press
'preview'.
- Sometimes getting a "The submitted form was invalid. Try submitting again." error message when trying to use the embedded image tag function.
- Minor cosmetic stuff like no user 'location / mood' and no smilies in the 'quick reply' section.
|
|
|
Melgar
Anti-Spam Agent
Posts: 2004
Registered: 23-2-2010
Location: Connecticut
Member Is Offline
Mood: Estrified
|
|
@chemplayer I like that hovering over the buttons will show you the context, so you don't even need to use them.
There might be some sort of restriction on new users including images. I was hoping one of the admins would take a look at the admin tools over
there.
Right now, I'm trying to set up a better attachment system. If it works, you guys are really gonna like it. 
The first step in the process of learning something is admitting that you don't know it already.
I'm givin' the spam shields max power at full warp, but they just dinna have the power! We're gonna have to evacuate to new forum software!
|
|
|
streety
Hazard to Others
Posts: 110
Registered: 14-5-2018
Member Is Offline
|
|
I agree with Tsjerk on this.
One area that might need some extra testing is search. I have in the past been a member on a phpBB forum where the search became unusable due to the
size of the forum. It was probably smaller than SM. I actually built a dedicated search engine just for the forum to replace the default.
That was back in 2009-2010 so hopefully no longer an issue but something to check.
|
|
|
sodium_stearate
Hazard to Others
Posts: 255
Registered: 22-4-2011
Location: guard duty at the checkpoint
Member Is Offline
Mood: No mask.
|
|
I honestly do not understand why anything needs to
be done to this place.
It seems to all work very well just the way it presently is.
Why attempt to fix it, if it's not broken?
Again, my suggestion is simply to leave well enough alone!
"Opportunity is missed by most people
because it is dressed in overalls and it
looks like work" T.A. Edison
|
|
|
streety
Hazard to Others
Posts: 110
Registered: 14-5-2018
Member Is Offline
|
|
How would we know if it is broken or not? For any security issues any issues will only become obvious after the site is hacked. Far better to be
proactive.
One approach would be to just patch this site whenever an issue is discovered. There are a small number of people able to do that though.
By moving to a more popular forum software there is a larger pool of people watching for issues and able to fix them.
It might be this site is fine. No security vulnerabilities to be found. However, a quick look at the html source for just this page does not fill me
with confidence.
|
|
|
sodium_stearate
Hazard to Others
Posts: 255
Registered: 22-4-2011
Location: guard duty at the checkpoint
Member Is Offline
Mood: No mask.
|
|
Kind of sounds to me like making more work
for one's self (and then later on, the users).
Sounds to me like inventing possible problems simply
as an excuse to change the whole works here at the forum.
I say, leave it alone! It works. It works well.
Why on earth are people these days so eager to always
change things just for the sake of change? Take something
that works well and is highly reliable, and then replace it
with some piece of junk "cloud" software deal that probably
does not work worth a hang....
That is my impression of it.
Possibly the present software on this forum works
just a little *too* well?
Maybe having it work well does not quite suit some
people around here.
Maybe having something that is all hosed up and does not
work well at all is somehow preferable?
Seriously. Just leave it alone.
"Opportunity is missed by most people
because it is dressed in overalls and it
looks like work" T.A. Edison
|
|
|
| Pages:
1
2
3
4 |
![[*]](images/xpblue/default_icon.gif){kind=icon}
