hi,
check out http://loci.cs.utk.edu/ for a way of distributing huge textures using university servers!!!.
you can download tools from http://loci.cs.utk.edu/modules.php?name ... d&cid=7_53
for example a 700mb linux distribution can be downloaded in 3 hours on cable
the project is free for public use.
Distributing large textures really fast
-
Guest
Do you need a BSD or Linux ISO image, but you do not want to wait for an hour to get it using HTTP or FTP? If so, Logistical Networking may help. Like mirroring, we store multiple copies of each of these ISO images in several places in the US. Each copy is striped over multiple servers. For each ISO, we have an exNode that contains the locations of all the pieces. Unlike HTTP or FTP mirroring where the user must choose a mirror, our download clients automatically try to retrieve pieces of the file from all available servers. In doing so, they automatically load-balance over all the servers to achieve the highest throughput available. Using exNode-based Logistical Networking tools, you can cut the download time down to 5-10 minutes if you are connected to a research network such Internet2 and ESnet.
Currently, we have two exNode-compatible download clients. The Logistical Distribution Network (LoDN) client is a Java Web Start application. In combination with the LoDN environment, the LoDN client provides an integrated content distribution service. The LoDN Java clients are still in beta and may take a minute or two to get started. The other client tools are the LoRS tools. These are C based command line tools and libraries. We also provide a Tcl/Tk GUI for easier use. The LoRS tools may be more reliable when downloading files.
Currently, we have two exNode-compatible download clients. The Logistical Distribution Network (LoDN) client is a Java Web Start application. In combination with the LoDN environment, the LoDN client provides an integrated content distribution service. The LoDN Java clients are still in beta and may take a minute or two to get started. The other client tools are the LoRS tools. These are C based command line tools and libraries. We also provide a Tcl/Tk GUI for easier use. The LoRS tools may be more reliable when downloading files.
-
Guest
What are the LoRS Tools for?
The LoRS tools give ANYONE (Internet2 connected or not) read/write access to the unused storage space on the L-Bone (Go to the L-Bone listing to check available locations and storage). The L-Bone (more) is a collection of IBP (more) depots spread across the globe. The distribution of Linux using this infrastructure is to us an experiment, and to you, the chance to get your ISO in about 5-10 minutes. Our intention is to upload multiple copies of the ISOs into the L-Bone using the lors_upload tool. These copies will be geographically dispersed and broken in to smaller pieces. Once the copies are uploaded, the tool generates the exNode (more) pseudo-file. These exNodes will be available from this page for download with names like [distribution_name].iso.xnd. Once you have the tools installed, run lors_download using the exNode as an argument. Click here to see what an exNode looks like.
After installing these tools, you can upload your own files to the L-Bone and then send the exNode to others who would be interested in having a copy of the file. During the upload process you can determine how long your uploads reside in the L-Bone and even which depots to use based on locality/proximity (like ZIP codes). Click here for the L-Bone location API. The depots themselves also determine how long an allocation is allowed to exist. After the allocation time (user or depot determined) expires, the data is erased from the L-Bone.
Many people, including researchers in physics and bio-informatics, are already putting this storage space to good use by making large datasets easily available for their research communities. Others are using it to provide streaming video and audio. What do you plan to do with it? Tell us about it.
The LoRS tools give ANYONE (Internet2 connected or not) read/write access to the unused storage space on the L-Bone (Go to the L-Bone listing to check available locations and storage). The L-Bone (more) is a collection of IBP (more) depots spread across the globe. The distribution of Linux using this infrastructure is to us an experiment, and to you, the chance to get your ISO in about 5-10 minutes. Our intention is to upload multiple copies of the ISOs into the L-Bone using the lors_upload tool. These copies will be geographically dispersed and broken in to smaller pieces. Once the copies are uploaded, the tool generates the exNode (more) pseudo-file. These exNodes will be available from this page for download with names like [distribution_name].iso.xnd. Once you have the tools installed, run lors_download using the exNode as an argument. Click here to see what an exNode looks like.
After installing these tools, you can upload your own files to the L-Bone and then send the exNode to others who would be interested in having a copy of the file. During the upload process you can determine how long your uploads reside in the L-Bone and even which depots to use based on locality/proximity (like ZIP codes). Click here for the L-Bone location API. The depots themselves also determine how long an allocation is allowed to exist. After the allocation time (user or depot determined) expires, the data is erased from the L-Bone.
Many people, including researchers in physics and bio-informatics, are already putting this storage space to good use by making large datasets easily available for their research communities. Others are using it to provide streaming video and audio. What do you plan to do with it? Tell us about it.
-
Guest
-
Guest
-
Guest
-
Guest
It would be helpfull if you would register for this forum, or at least use a temporary username, so that it is obvious which posts belong to whom and whom one adresses in an answer.
My biggest texture pack so far is an extended blue marble with nights- and specs-textures and an uncompressed size of 1.1 GB. My biggest self made is 450 MB uncompressed, and not yet released.
I think people will try and use the LORS network if they have big distributions. Just be patient and give them some time. I have just managed to get 2GB of fast webspace, so it's not so urgend for me. Perhaps Chris will put the next release up there to buffer the access to 'shatters.net' - but he also did upgrade recently, so the load problem may be also not too critical here.
maxim
My biggest texture pack so far is an extended blue marble with nights- and specs-textures and an uncompressed size of 1.1 GB. My biggest self made is 450 MB uncompressed, and not yet released.
I think people will try and use the LORS network if they have big distributions. Just be patient and give them some time. I have just managed to get 2GB of fast webspace, so it's not so urgend for me. Perhaps Chris will put the next release up there to buffer the access to 'shatters.net' - but he also did upgrade recently, so the load problem may be also not too critical here.
maxim
-
t00fri
- Developer
- Posts: 8772
- Joined: 29.03.2002
- Age: 22
- With us: 22 years 7 months
- Location: Hamburg, Germany
Anonymous wrote:I started this discussion.
The point i'm trying to get across is if someone creates a large virtual texture e.g. 64k Earth this can be a 500mb file. Instead of putting it on a SLOW ftp server they can upload it to lors. Then everyone else can download it really FAST.
It seems to me that you are advocating a massive misuse of expensive university server bandwidth?
Bye Fridger
-
Guest
LoRS not misuse
LoRS is the product of the Logistical Networking research project which explores the use of storage, bandwidth and processing resources in advanced networking applications. We share project resouces, some of which are located at Universities, with communities of early adopters in order to support of our own research.
We encourage you to use those resources appropriately in support of your work - our research is in the monitoring and managing such use. Storing and distribuing files a few GB in size is certainly no strain on our resources. Bring it on!
As with Internet resources, it is a misuse of Logistical Networking resources to circumvent our controls on the level of use of resources by individuals, or to distribute content illegally. Please don't do these things.
PS. I am a different guest from the one who introduced LoRS to this list, but we probably work together.
We encourage you to use those resources appropriately in support of your work - our research is in the monitoring and managing such use. Storing and distribuing files a few GB in size is certainly no strain on our resources. Bring it on!
As with Internet resources, it is a misuse of Logistical Networking resources to circumvent our controls on the level of use of resources by individuals, or to distribute content illegally. Please don't do these things.
PS. I am a different guest from the one who introduced LoRS to this list, but we probably work together.
-
rich
Hi - i started this thread, im called rich btw.
Quote from the LORS site "Many people, including researchers in physics and bio-informatics, are already putting this storage space to good use by making large datasets easily available for their research communities.".
As Celestia is certainly a project to further scientific understanding it comes under this category. Therefore we would NOT be misusing this resource by distributing virtual textures!!
Quote from the LORS site "Many people, including researchers in physics and bio-informatics, are already putting this storage space to good use by making large datasets easily available for their research communities.".
As Celestia is certainly a project to further scientific understanding it comes under this category. Therefore we would NOT be misusing this resource by distributing virtual textures!!
-
t00fri
- Developer
- Posts: 8772
- Joined: 29.03.2002
- Age: 22
- With us: 22 years 7 months
- Location: Hamburg, Germany
Rich,rich wrote:Hi - i started this thread, im called rich btw.
Quote from the LORS site "Many people, including researchers in physics and bio-informatics, are already putting this storage space to good use by making large datasets easily available for their research communities.".
As Celestia is certainly a project to further scientific understanding it comes under this category. Therefore we would NOT be misusing this resource by distributing virtual textures!!
thanks for making yourself slightly less anonymous
I have serious doubts concerning your statements. As I know too well from the regulations of my own laboratory, we have very strict constraints as to using our resources for non professional purposes. It is certainly far from sufficient to come up with Celestia being of "general scientific interest".
Within Europe at least , universities are running on a much tighter budget than we are and correspondingly are even less tolerant...
In my lab, every transfer is logged and examined by our security team. There is a 100% "allergy" towards anything smelling even remotely of file sharing...
There is a concrete threat that people being invoked in file sharing transactions will be fired right away...
Most research institutions adhere to a similarly strict philosophy.
Are you telling us that we should just ignore this and stay cool?
Bye Fridger
Fridger,
According to its home page at the University of Tennessee, LoCI seems to be a research project in the logistics of very large-scale distributed file and computing systems. Its PIs are at the University of Tennessee, and it's supported by the U.S. Department of Energy and the U.S. National Science Fundation. In addition to providing "personal" encrypted storage, they seem to be trying to find uses which will provide a severe stress test of its servers and of Internet2.
The current test applications that they're directly supporting are
1. downloading BSD and Linux CDs
2. recording and downloading TV programs.
3. transferring multi-gigabyte files as Mail messages
4. distributed file management, including automatic relocation to different servers when storage leases expire.
Browsing their pages, I have not yet found any explicit statement about "who may use LoCI". The description of the CD distribution application does say something about having an Internet2 connection.
Since it's a research project, it's hard to tell just how long it might last, although it seems to have been under development for several years.
According to its home page at the University of Tennessee, LoCI seems to be a research project in the logistics of very large-scale distributed file and computing systems. Its PIs are at the University of Tennessee, and it's supported by the U.S. Department of Energy and the U.S. National Science Fundation. In addition to providing "personal" encrypted storage, they seem to be trying to find uses which will provide a severe stress test of its servers and of Internet2.
The current test applications that they're directly supporting are
1. downloading BSD and Linux CDs
2. recording and downloading TV programs.
3. transferring multi-gigabyte files as Mail messages
4. distributed file management, including automatic relocation to different servers when storage leases expire.
Browsing their pages, I have not yet found any explicit statement about "who may use LoCI". The description of the CD distribution application does say something about having an Internet2 connection.
Since it's a research project, it's hard to tell just how long it might last, although it seems to have been under development for several years.
Selden
-
rich
t00fri - Did you actually read the whole thread? or look at the website!!
According to the guy who WORKS there "Storing and distribuing files a few GB in size is certainly no strain on our resources. Bring it on!"
Also this project has terabytes of space and loads of bandwidth distributed over thousands of universities all over the world - look at the map on the website.
It IS open for public use and they WANT people to use it (legally). Internet2 is not required.
Maybe you should contact them if you are in any doubt.
According to the guy who WORKS there "Storing and distribuing files a few GB in size is certainly no strain on our resources. Bring it on!"
Also this project has terabytes of space and loads of bandwidth distributed over thousands of universities all over the world - look at the map on the website.
It IS open for public use and they WANT people to use it (legally). Internet2 is not required.
Maybe you should contact them if you are in any doubt.
-
t00fri
- Developer
- Posts: 8772
- Joined: 29.03.2002
- Age: 22
- With us: 22 years 7 months
- Location: Hamburg, Germany
OK, I had a more detailed look at the LoCI site.
Given the worldwide IBP network of presently ~350 servers (142 not working), a total r/w storage of ~26000 GB is made available to the public at this time.
That's nice and may find a number of applications as long as the project is running. Research networks like Internet2 or ESnet are not needed at that stage.
However, it seems to me that the advocated highly accelerated download of Celestia textures, for example, via exNode-based Logistical Networking tools does effectively require the use of Internet2 or similar high-speed network resources. This would be against the policy of most research institutions connected via such research networks, since their bandwidth is apparently misused...
Without such fast (100+Mb/sec) research networks, an optimized load balancing is still performed among the participating servers, but I guess for slower networks that effect is not so dramatic...?
For example: With a slow modem download there is only
1 thread and not much to be gained. With a really fast research network (100+ Mb/sec), the number of threads may be taken 10-20 and then the speedup can be really large. With cable/SL/T1 the number of threads would be typically 3.
Right or wrong?
Bye Fridger
Given the worldwide IBP network of presently ~350 servers (142 not working), a total r/w storage of ~26000 GB is made available to the public at this time.
That's nice and may find a number of applications as long as the project is running. Research networks like Internet2 or ESnet are not needed at that stage.
However, it seems to me that the advocated highly accelerated download of Celestia textures, for example, via exNode-based Logistical Networking tools does effectively require the use of Internet2 or similar high-speed network resources. This would be against the policy of most research institutions connected via such research networks, since their bandwidth is apparently misused...
Without such fast (100+Mb/sec) research networks, an optimized load balancing is still performed among the participating servers, but I guess for slower networks that effect is not so dramatic...?
For example: With a slow modem download there is only
1 thread and not much to be gained. With a really fast research network (100+ Mb/sec), the number of threads may be taken 10-20 and then the speedup can be really large. With cable/SL/T1 the number of threads would be typically 3.
Right or wrong?
Bye Fridger