Future of the Forum

[Donn]

Which of those Digital Ocean plans will be used? And how much does firewalling cost? (I assume this is anti-DDOS, rather than mere iptables.)

 

[paulhutch]

Here's GoDaddy's pricing. For $15/month, unlimited everything. (This is shared hosting, but dedicated isn't that much more expensive IIRC.)

Dedicated hosting is ten times more expensive than shared hosting at GoDaddy: https://www.godaddy.com/hosting/dedicated-servers.aspx?ci=9014

 

[icerat]

Which of those Digital Ocean plans will be used? And how much does firewalling cost? (I assume this is anti-DDOS, rather than mere iptables.)

How long is a piece of string? It's a balance between having to do the work ourselves and outsourcing.

We're going to kick off with the 8Gb/4core/80Gb SSD hosting with live backup capability, out of NYC. It's going to be a little tight initially when we're running two databases and backups but once the transition is done it should do the job fine. Basic Sucuri firewalling, and at some stage migrating to XenForo with Enhanced Search and Resource Manager.

 

[Donn]

Thanks for the info icerat. I make that to be about $1080 for a year. Do you think we can have a thread, on the new system, that reveals the details (who, where, what) and the costs on an ongoing basis - to help us decide how much to contribute and how often?

Is the account at Digital Ocean (etc.) going to be in your/TribeTech's name?

Aside: Up to this point, I had thought your own IT company's (in Sweden) servers would be doing the hosting.

 

[quadraginta]

This forum will go down in history as probably the only internet forum outside the former Soviet Union nations that ever required people's real names to register.

Probably not.

I've know of others, bigger than JREFF, that went beyond that to not accepting registrations with cloud based email addresses like @live or @gmail.

You had to use something like an email address from your ISP or something, or spend time authenticating yourself with the board admins.

 

[dasmiller]

Probably not.

I've know of others, bigger than JREFF, that went beyond that to not accepting registrations with cloud based email addresses like @live or @gmail.

ouch! I've been migrating *away* from ISP-specific email addresses because they become problematic whenever I want to change ISPs. I now consider my gmail address to be the "real" one.

 

[Henry Bannister]

Be quiet!

Ah, now we see the violence inherent in the system.

 

[quadraginta]

ouch! I've been migrating *away* from ISP-specific email addresses because they become problematic whenever I want to change ISPs. I now consider my gmail address to be the "real" one.

Yeah. Me too.

The sites I've found that do that generally let you change your email address afterwards, and like I said, you can get around the requirement with some negotiation, but that's the default.

I guess it helps keep down the bots somehow, or something. I have no idea if it really matters. Sites like XDA and BleepingComputers don't bother with anything like that, and they've got a lot more network savvy on board than the boards I'm talking about.

Probably some owner/admin "clever idea" that they read about somewhere.

 

[icerat]

Thanks for the info icerat. I make that to be about $1080 for a year. Do you think we can have a thread, on the new system, that reveals the details (who, where, what) and the costs on an ongoing basis - to help us decide how much to contribute and how often?

It's going to be a while before things have settled down and we have a handle on performance etc, but I can't see why not.

Is the account at Digital Ocean (etc.) going to be in your/TribeTech's name?

Yes.

Aside: Up to this point, I had thought your own IT company's (in Sweden) servers would be doing the hosting.

We're using servers here in Sweden and in the UK, but most of the JREF traffic is in North America, with UK and Oz coming in next. US makes the most sense performance wise. Thought about CDN's but I've had hit and miss experiences with the lower price ones. NYC should give reasonable access for Europe and US and is better than Europe for the Aussies et al.

 

[Donn]

Thanks again.

 

[Sherman Bay]

How big was the board, and how many hours a month did you spend running it?

Not big, and one of the reasons I discontinued it was the hassle of administration (my time). Keeping things in line, wiping out spammers, answering questions like "how do I post a reply?" -- these became increasingly unrewarding. Admin costs here at JREF, unless mods are paid, isn't a factor in where to host it. My point is even if my board had ballooned by a factor of 1000, the hosting cost (software/hardware) would not have increased much, if at all.

Even if the quantity of videos I made available had ballooned by a factor of 100, my costs would not have increased significantly. (I moved my video to YouTube as that site's quality became better every year.)

Anyway, we kind of have a bit of knowledge in this area and I and some others have spent a deal of time looking at the requirements of this site. Storage of 500 videos is in fact easier to deal with than a database with nearly 10000000 posts. I can guarantee you that we do not want to put it on GoDaddy shared hosting.

And nowhere did I suggest that shared hosting was the answer, only a benchmark.

Let's do some rough math. 500 videos at 2GB/per. Streaming (that's continuous, or nearly so) data transmission at 1-2Mb/sec, for several videos at once to different viewers, dependent upon how many, of course, but viewers typically watched complete 1 to 2 hour shows each. Contrast that with sending a few K of text data for a post or a page of a thread -- discontinuous -- multiplied by how many users simultaneously posting and/or reading, and the video starts to look like a more of a memory and data hog than a text-based message board.

Since I won't be paying for this experiment except as a user -- should you decided to charge for membership -- it really doesn't matter to me. If whoever takes financial responsibility for this site can afford it, that's great.

 

[icerat]

Not big, and one of the reasons I discontinued it was the hassle of administration (my time). Keeping things in line, wiping out spammers, answering questions like "how do I post a reply?" -- these became increasingly unrewarding. Admin costs here at JREF, unless mods are paid, isn't a factor in where to host it. My point is even if my board had ballooned by a factor of 1000, the hosting cost (software/hardware) would not have increased much, if at all.

Even if the quantity of videos I made available had ballooned by a factor of 100, my costs would not have increased significantly. (I moved my video to YouTube as that site's quality became better every year.)

And nowhere did I suggest that shared hosting was the answer, only a benchmark.

Let's do some rough math. 500 videos at 2GB/per. Streaming (that's continuous, or nearly so) data transmission at 1-2Mb/sec, for several videos at once to different viewers, dependent upon how many, of course, but viewers typically watched complete 1 to 2 hour shows each. Contrast that with sending a few K of text data for a post or a page of a thread -- discontinuous -- multiplied by how many users simultaneously posting and/or reading, and the video starts to look like a more of a memory and data hog than a text-based message board.

It's the simultaneous text writing and retrieval part that's intensive, both on CPU and HDD. Size per se isn't really an issue these days. With video streaming it depends on whether you're doing the decoding work on the server side or client side.

 

[temporalillusion]

My point is even if my board had ballooned by a factor of 1000, the hosting cost (software/hardware) would not have increased much, if at all.

Going from 10 concurrent users to 10,000 concurrent users (or 100 to 100,000) would still only cost $15 or a little more a month for hardware??

That's amusing.

 

[barehl]

I bought no hardware and the only software was vBulletin, about $250 for a perpetual license; a little different if you want updates on a regular basis.

The hosting company has the Internet access, backup and storage, and it is getting cheaper all the time. 1.544 mb/sec -- you gotta be kidding. That's so 1970's. I don't know the size of randi.org's message board database, but I'll bet it pales in comparison to the 500 full-length videos I had stored on my site once. We're not talking about a challenging or expensive site -- this is trivial in traffic, processing and storage.

Here's GoDaddy's pricing. For $15/month, unlimited everything. (This is shared hosting, but dedicated isn't that much more expensive IIRC.

I wish I could say something positive about your post but I really can't. If you recall, I mentioned a ways back that if not for the archive, this forum could be hosted for free on FreeForums.org. So, you should have understood then that I was familiar with low cost options. There are numerous options like GoDaddy that are a little (but not much) better than the free options. You didn't pay for a server with $15/month. You paid for a piece of one rack mounted server board in a chassis in a room full of rack-mount servers. You paid for a piece of the Ethernet bandwidth on that one server. You paid for shared use of the memory on that board and some small fraction of the RAID attached to it. Your entire year's rental would not pay for that single sever board, much less the RAID, internet access and electricity. That you think this is somehow equivalent to a dedicated server or dedicated bandwidth not exactly rational.

 

[barehl]

My point is even if my board had ballooned by a factor of 1000, the hosting cost (software/hardware) would not have increased much, if at all.

This is not a realistic claim. I'm baffled that you seem to think that everything is free.

Even if the quantity of videos I made available had ballooned by a factor of 100, my costs would not have increased significantly. (I moved my video to YouTube as that site's quality became better every year.)

YouTube has built-in advertizing. It's only free in the same way that broadcast television or radio is free. It's actually paid for by advertisers.

And nowhere did I suggest that shared hosting was the answer, only a benchmark.

Then why did you compare shared hosting to my description of common costs?

Let's do some rough math. 500 videos at 2GB/per. Streaming (that's continuous, or nearly so) data transmission at 1-2Mb/sec, for several videos at once to different viewers, dependent upon how many, of course, but viewers typically watched complete 1 to 2 hour shows each. Contrast that with sending a few K of text data for a post or a page of a thread -- discontinuous -- multiplied by how many users simultaneously posting and/or reading, and the video starts to look like a more of a memory and data hog than a text-based message board.

The actual cost of bandwidth is dependent on the excess capacity at the local switching station and the distance between your facility and theirs. If they have lots of extra capacity then you can get a good deal because unused capacity makes them no money. And, if you are physically close then that is good too.

However, even with plenty of bandwidth, you are still going to run out somewhere. With numerous users the limit would be hard drive seek time. Unless the entire database is loaded into memory, you'll have to search the hard drive for every read, every new post, and every search. Each one of these has to be loaded into memory and new posts have to be written back out. There's no way to speed this up unless you go to some kind of distributed system, preferably load balanced.

 

[Spyke]

If nothing else, I'm learning a ******** of technical stuff from this Thread!

 

[Mick West]

We're going to kick off with the 8Gb/4core/80Gb SSD hosting with live backup capability, out of NYC. It's going to be a little tight initially when we're running two databases and backups but once the transition is done it should do the job fine. Basic Sucuri firewalling, and at some stage migrating to XenForo with Enhanced Search and Resource Manager.

That should be fine. I run a similar forum (http://metabunk.org) with comparable traffic, but much smaller database on 4GB Linode (now the same price as Digital Ocean equivalents). I use Cloudflare Pro ($5/Month) for firewall and CDN. Xenforo with some tweaks. Handles 2000 active users (as measured by Google Analytics) - which I get very rarely in spikes.

Edit: Analytics "active user" is for a 5 minute window. The default VB active user is 15 minutes. So JREF's peak of 2,222 likely represents <800

 

[icerat]

That should be fine. I run a similar forum (http://metabunk.org) with comparable traffic, but much smaller database on 4GB Linode (now the same price as Digital Ocean equivalents). I use Cloudflare Pro ($5/Month) for firewall and CDN. Xenforo with some tweaks. Handles 2000 active users (as measured by Google Analytics) - which I get very rarely in spikes.

Edit: Analytics "active user" is for a 5 minute window. The default VB active user is 15 minutes. So JREF's peak of 2,222 likely represents <800

Main challenge is the forum DB alone (not file system) is 18GB and we're going to need to run two versions of it while handling the privacy issues, so that's 36GB of our 80GB gone before we even talk about OS, file system (forum images, logs web, etc) and any local DB backups.

We should be fine with off system backups and once we can shutdown the old DB things it should be plenty. If necessary we can bump up the service size for a month or so.

I've been testing CloudFare, had a wiki speed up considerably and a wordpress blog slow down considerably, so still need some tweaking to optimize it.

 

[Mick West]

Main challenge is the forum DB alone (not file system) is 18GB and we're going to need to run two versions of it while handling the privacy issues, so that's 36GB of our 80GB gone before we even talk about OS, file system (forum images, logs web, etc) and any local DB backups.

We should be fine with off system backups and once we can shutdown the old DB things it should be plenty. If necessary we can bump up the service size for a month or so.

I'd recommend doing that. I don't know about DO, but Linode lets you play around with the size of things without any real charges beyond hourly usage. The larger and faster server will be very useful during transition.

I've been testing CloudFare, had a wiki speed up considerably and a wordpress blog slow down considerably, so still need some tweaking to optimize it.

On an unloaded server it's quite possible for Cloudflare to slightly slow things down. It works best under load. It will be incredibly helpful in offloading the loading of static resources for first time or infrequent visitors. Properly configured with Xenforo caching it cuts down 30+ requests to 1. It should still show significant improvement with vanilla VB too.

 

[steenkh]

What are we going to do about the JREF specific sub-forums? Leave them as they are? Have them are archive only forums

I certainly do not hope they will be deleted.