Wretched forum performance

[timcrews]

Hello:

I know the forum occasionally has times when it is slow, but for the last four days it has _always_ been slow, and often so slow that minutes go by before my screen refreshes.

Today I needed to make three posts, and it took almost a half an hour. I lost data that I typed in two different posts, because of a PHP "too many connections" or "couldn't connect to database" error, before I learned to copy everything I had typed to the clipboard before submitting my message.

Thanks for letting me get that off of my chest.

Tim

[RND]

yeah ive noticed a definate slow down in the past week although it doesnt appear too bad today. I even had the odd dns error last week.

[selden]

Part of the problem is due to heavy loads on Shatters.net and another part is due to congestion in the networks between your system and Shatters. On a Windows system you can use the "DOS" command

Code: Select all

tracert www.shatters.net

to find out where the slowdown is. The linux equivalent is spelled slightly differently: traceroute

Nothing much can be done about overall network congestion, but it seems to me that Shatters' performance might be improved if many of the files could be moved to another server. I think many of them could justifiably be made available on SourceForge, for example. I fear the image galleries may be a significant part of the load, though, and those probably couldn't be moved.

Patience is a virtue.
Or so they say.

[RND]

Tracing route to http://www.shatters.net [63.224.48.65]

over a maximum of 30 hops:



1 * * * Request timed out.

2 93 ms 82 ms 82 ms gb-ldk-c-r2-ge60.inet.business.ntl.com [80.1.208.2]

3 93 ms 85 ms 83 ms gb-ldk-c-r1-pos00.inet.ntl.com [80.1.209.161]

4 93 ms 82 ms 83 ms pop-ic-1-ge-010-0.inet.ntl.com [62.253.184.249]

5 93 ms 81 ms 87 ms pop-bb-a-ge-700-0.inet.ntl.com [62.253.184.242]

6 92 ms 85 ms 82 ms so-7-0.hsa2.London1.Level3.net [212.113.14.129]

7 90 ms 83 ms 80 ms so-4-0-0.mp2.London1.Level3.net [212.187.131.125]

8 184 ms 186 ms 2934 ms so-1-0-0.bbr2.NewYork1.Level3.net [212.187.128.153]

9 1133 ms 236 ms 151 ms ge-7-0-0.edge1.NewYork1.Level3.net [64.159.4.150]

10 158 ms 153 ms 150 ms ewr-brdr-02.inet.qwest.net [205.171.1.97]

11 163 ms 150 ms 147 ms ewr-core-02.inet.qwest.net [205.171.17.129]

12 190 ms 182 ms 177 ms chi-core-01.inet.qwest.net [205.171.8.229]

13 190 ms 176 ms 175 ms chi-core-02.inet.qwest.net [205.171.20.170]

14 296 ms 563 ms 2235 ms cer-core-01.inet.qwest.net [205.171.205.34]

15 278 ms 185 ms 182 ms cer-core-03.inet.qwest.net [205.171.139.6]

16 251 ms 248 ms 243 ms tkw-core-01.inet.qwest.net [205.171.205.113]

17 255 ms 245 ms 275 ms tukw-agw1.inet.qwest.net [205.171.11.90]

18 256 ms 244 ms 248 ms 63.231.10.68

19 267 ms 261 ms 260 ms 63.224.48.70

20 268 ms 260 ms 262 ms 63.224.48.65



Trace complete.

performance does seem to have improved in the last day or so.[/quote]

[selden]

RND,

The delay times in your traceroute seem to have a gradual increase for each router. I suspect it's about as good as you'll get.

When Shatters.Net is overloaded, the last two lines usually are greater than 1000 ms.

[chris]

I'm thinking very seriously about moving shatters.net to another provider that can offer me about 50% more upstream bandwidth than I have now. Access to the site would be disrupted for a couple days, but it'd be worth the hassle. For those who are wondering where the bandwidth is going, here are the top few files downloaded this week, according to my HTTP stats program:

Code: Select all

%Reqs %Byte  Bytes Sent  Requests   Archive Section
----- ----- ------------ -------- |------------------------------------
 0.52 20.46  13784861157     2069 | /celestia/files/celestia-win32-1.3.1-1.exe
 0.07  8.84   5955048350      287 | /~impulse/Mars_Central/8kRedMars-05-12-03.zip
 0.07  6.88   4632479108      295 | /~impulse/Earth_Central_v2/Texture_downloads/Realistic-Earth16kdds.zip
 0.09  5.89   3969379605      359 | /~t00fri/textures/8k/dds/redmars8k-bmp-um-redcorr.dds.zip
 0.07  5.87   3956433682      277 | /~impulse/NASA/Frank/Mars/New-Mars-Clouds.zip
 0.10  3.67   2469150820      388 | /~t00fri/textures/8k/dds/earth8k-BM-col-bmp-um.dds.zip
 0.03  3.35   2255788050      115 | /~impulse/Earth_Central/downloads/Earth16kDDS-121203.zip
 0.20  2.66   1793481138      813 | /celestia/docs/CelestiaUsersGuide-1.3.1.doc
 0.02  2.44   1643290977       71 | /~rassilon/V380.zip
 0.04  2.25   1516929788      144 | /~impulse/Misc/Mars.zip
 0.01  2.17   1461625085       51 | /~rassilon/muphrid.zip
 0.02  1.86   1251916605       89 | /~impulse/Earth18kBCE/Finalpix/Earth18000BCE.zip
 0.01  1.44    971565709       21 | /~praesepe/files/Moon16K-Single.zip
 0.01  1.27    853107477       37 | /~rassilon/gc_tex_v2.zip
 0.01  1.25    838900999       24 | /~praesepe/files/Moon16KVT.zip
 0.05  1.17    789702582      197 | /~t00fri/textures/8k/jpg-png/redmars8k-bmp-um-redcorr.jpg
 0.05  1.15    772571166      220 | /~t00fri/textures/2k/jpg-png/earth-clouds.png
 4.27  1.10    741053621    17143 | /forum/viewtopic.php
 0.02  1.09    733441335       74 | /~jim/files/moon_8k_color_v2_VT_dds.zip
 0.01  1.06    714761540       26 | /~rassilon/alrescha.zip
 0.06  0.99    664664461      249 | /~t00fri/textures/8k/jpg-png/earth8k-BM-col-bmp-um.jpg
 0.01  0.82    550157464       24 | /~impulse/Earth_Central/downloads/clouds-121303-8kDDS.zip


A surprising number of people are downloading Celestia 1.3.1 right from shatters.net instead of SourceForge. I need to figure out who's linking to it, since it's responsible for 20% of my traffic. The rest of the traffic is mostly from the large add-ons that I host on shatters.net.

--Chris

[Don. Edwards]

Well I can be blamed for 6 of the offending files. But I am going to remove my old 16k texture so that should help just a little. Someone must have it hyperlinked somewhere.

Don. Edwards

[selden]

Chris,

I have a link to the "files" directory on my site, with the comment "Where Chris provides the latest versions of Celestia along with some addon components."

I have an awful feeling that a lot of accesses come by way of that.

I want to keep the link so people can find the prereleases and addons, but obviously that comment has to be changed -- I'll add "for testing only".

I'll also add yet another link to sourceforge that's right next to that link.

*sigh*

[timcrews]

Hello:

Is it impolite of me to ask how shatters.net is paid for? What amount of voluntary monetary contribution would it take on a monthly basis to host the forum on a different internet connection? Does the forum itself use much network bandwidth? Is the shatters.net server ever bogged down, or is the bottleneck always in the network connection?

Tim

[AuReality]

Hello:

Is it impolite of me to ask how shatters.net is paid for? What amount of voluntary monetary contribution would it take on a monthly basis to host the forum on a different internet connection? Does the forum itself use much network bandwidth? Is the shatters.net server ever bogged down, or is the bottleneck always in the network connection?

Tim

A valid question in my opinion.

Another proposal: Chris, you might want to consider to separate the forum from the download files (Celestia, add-ons, textures). That way the forum is not bothered by the high download traffic.

Paul

[Paolo]

Perhaps my intervention should be inapprorpiate but I would like to say 2 words.

Many times I was tempted to submit the same questions reported by Tim in the previous post. Many times I was discouraged by various Development team interventions about the free nature of Celestia.

But I think that actually voluntary monetary contribution is an extremely important issue. Also because when the add-on manager will be implemented a lot of web space an bandwidth will be required somewhere.

I don't know if Chris is actually sponsored in some manner, but I surely would like to contribute with some kind of donation.

Bye - Paolo.

[don]

Chris
* You can add my vote to have the forum software on a non-loaded server.

* Maybe you should not post final .exe versions to shatters.net, but only pre-releases?

* You can also count on me for a donation! The SourceForge site now *has* a donation system set-up, but I'm not sure exactly how it works, or how much goes to SF and how much to the Project you want to donate to. Might be worth looking into for Celestia <smile>.

Selden
You might want to check the "latest files" page link wording in the User's FAQ also, and change if necessary.

Cheers,

-Don G.

[AuReality]

You can also count on me for a donation.

Paul

[selden]

Don,

The "preliminary user's faq" is the only one I have any control over. It doesn't mention where to get the current version at all. I'll add that as Q/A #0.

[Harry]

Code: Select all

%Reqs %Byte  Bytes Sent  Requests   Archive Section
----- ----- ------------ -------- |------------------------------------
 0.52 20.46  13784861157     2069 | /celestia/files/celestia-win32-1.3.1-1.exe


A surprising number of people are downloading Celestia 1.3.1 right from shatters.net instead of SourceForge. I need to figure out who's linking to it, since it's responsible for 20% of my traffic.

The only link to it I know of is one in the forums, but this probably wouldn't explain the large number.

BTW, for what period of time is this data?

However, one think which may be worth noting:
There have been 13GB in ~2000 requests, which is about 6 MB per request, while celestia-1.3.1-1 is about 12 MB. Some people are probably stopping their transfer before it's finished. But some are probably using download "accelerators" which use many partial transfers at once, increasing the number of requests - which is a Bad Idea (TM), specially so on (over-)loaded sites. And I've seen these tools download more data than necessary, too.

This makes the situation worse and is unfair for other users without accelerators. Of course simply restarting transfers is an entirely different matter, and a good thing So please use those tools with care and reduce the number of parallel connections (I assume that's configurable).

Harald

[don]

The "preliminary user's faq" is the only one I have any control over. It doesn't mention where to get the current version at all.

Hi Selden,

Sorry I wasn't more specific. I was referring to A3 b, but it looks like it is worded okay <smile>.

-Don G.

----------------------------
Q3:
Where can I get another version of Celestia that might work better?

A3:
a) Older versions of Celestia are available on SourceForge:
See http://sourceforge.net/project/showfile ... p_id=21302
v1.2.4 seems the most robust, but does not include many recent features.

b) Some "prerelease" versions of Celestia for Windows are available
on Shatters.net: See http://63.224.48.65/celestia/files/
The most recent "prerelease" (e.g. V1.3.1pre9 or later) may work for you, but may be even worse.
----------------------------

[don]

Today (Thu Jan 8 2004) between 1pm and 1:25pm MST, the following errors are displayed every 4 or 5 clicks while trying to navigate the forums.

-Don G.
________________________________________
Warning: Too many connections in /var/www/html/forum/db/mysql4.php on line 48

Warning: MySQL Connection Failed: Too many connections in /var/www/html/forum/db/mysql4.php on line 48

Warning: mysql_error(): supplied argument is not a valid MySQL-Link resource in /var/www/html/forum/db/mysql4.php on line 330

Warning: mysql_errno(): supplied argument is not a valid MySQL-Link resource in /var/www/html/forum/db/mysql4.php on line 331
phpBB : Critical Error

Could not connect to the database
________________________________________

[Paolo]

Today (Thu Jan 8 2004) between 1pm and 1:25pm MST, the following errors are displayed every 4 or 5 clicks while trying to navigate the forums.

-Don G.

I don't remember exaclty the period but I get the same problem.
It was the first time for me.

Bye - Paolo

[alphap1us]

I have a couple of ideas about how to increase the performance of the forum. The main problem seems to be the other files that are soaking up the bandwith on shatters.net. I think the best thing to do would be to move as many of the other files of this server as possible. I can think of a couple of ways of doing this. ibiblio.org has a lot of stuff and a lot of bandwidth, which I understand is being underused. Perhaps they would be willing to host some of the add-ons and textures, since celestia is "pretty cool" even to people that are not into astronomy and physics and ibilio is all about open-source and linux. This would probably require deciding on "official" higher resolution textures and add-ons so that they are not hosting nearly identical copies of the same textures. I notice a lot of people reduplicating texture work on the most popular planets (Earth, Mars, Mercury.) I think deciding on the "best" texture in each resolution would be a good idea regardless of hosting since it would be easier for newbies to find the really good textures without searching thru a million different sites. Additionally, how hard would it be to upload some of the textures to the sourceforge site and host them there? I don't know what their policy is on hosting add-ons to programs that they host anyway, but I think it could be explored. One perhaps more difficult idea would be to package some textures together and make a torrent of them. Info can be found at bitconjurer.org/bittorrent but basically people share their upload bandwicth to let others download large files. Files can be hosted on a home computer without a static IP address and only the torrent file needs to hosted on a permanent server. Any people who already have the files and are willing to share the excess bandwidth could serve the files whenever the felt like it. I don't think this would be too hard to set up on shatters.net and I think it would take an extermely small amount of bandwith. I am not so hot at programming but I could contact ibiblio and ask them what they think, if someone how actually has some authority for the project wants to contact me and delegate some of it. What do people think?

[don]

Howdy alphap1us,

I'm always open to exploring other possibilities, but Chris might already have something in the works.

Chris?

I love ibiblio.org's tagline: "the public's library and digital archive".

Cheers,

-Don G.