Getting Started with ComputingComputing in the Hydrology Research group is done primarily on the UNIX operating system (FreeBSD). UNIX can be daunting to learn, but it is a powerful and flexible computing platform. To get started you might try reading O'Reilly's "Learning the UNIX Operating System." by Jerry Peek. It is available at the University bookstore.There are a few low power computers with Microsoft Windows installed for preparing documents using Office software. Office software is also available under UNIX. A listing of available computer systems is here.
Research ComputingThe large majority of processor time and disk space is used for research compute jobs. It is important since all resources are shared that you understand how to do this efficiently and in such a way that others can also do their work.Following are a couple of points to assist you: 1. CPU & MEMORY Keep an eye on your processes including how much CPU and memory they are using up. The "top" command is useful for this. It displays the processes which are using the most system resources, and how much CPU and memory they are using. Keep in mind that "top" itself consumes resources and should only be used to check on your process, not run constantly. Most systems are dual processors, so for these machines a steady load of 2 is okay. This means that on average 2 jobs are on the processors. When the load goes higher, a great deal of efficiency and compute cycles are lost to switching jobs on and off the processor. 2. DISK STORAGE Keep an eye on your disk space usage. Given the large models that the group works with, it is quite easy to consume vast amounts of disk space. The "du" utility is very useful for this. It can give a disk usage summary for files and directories. It is best to use the -k flag always, as some versions of UNIX do not report usage in kilobytes. Backup tapes, a CD-write and DVD-write drive are all made available for archiving files and removing them from active disk space. A guideline for how to lay out your files and directories as well as backup and restore considerations is available here. 3. NETWORK Systems are interconnected by 100Mbps (megabits/second) shared links. This works out to a theoretical maximum of about 13 Megabytes/second. In reality it is much less. We regularly deal with files from several Megabytes to more than a Gigabyte. This means: although it is possible, DO NOT RUN COMPUTE JOBS OVER THE NETWORK. All files for the process should be local to the machine. The capability of running jobs with remotely located files will be removed at some time in the future. To access other machines on the network, we have a couple of methods available. The primary method is SSH Secure Shell which allows secure encrypted access to other machines. From a Windows machine you can log in to a UNIX machine, or you can transfer files using SSH Secure FTP. From one UNIX machine to another you can log in and transfer files using ssh and scp or sftp. We also have nfs automount between UNIX machines allowing you to easily copy files between machines. As mentioned above DO NOT use this service to run compute jobs over the network. Lastly files can be accessed through SAMBA or Windows shared disks. This is the least reliable, least secure and least recommended method. 4. SOFTWARE We use a large variety of free UNIX software to do our work. There is also a limited amount of commercial software installed. In general, you can request that any free software package be installed and it will be installed quickly and available on all of our UNIX systems from our central server. Purchases of commercial software must be approved by Dennis and will take longer to aquire and install. Other Computing 1. WINDOWS SYSTEMS We have a limited number of Windows PCs available for shared use. As noted in the backups document do not keep ANY files on the Windows PCs that are of value to you. These systems are not backed up at all. These systems are a shared resource for working on papers and spreadsheets that require the Microsoft platform. Be aware that at certain times during the school year the systems will be very busy with people writing term papers etc. Please be considerate of others. On the UNIX systems we have StarOffice available which has some compatibility with Microsoft Office formatted documents. The compatability is not always perfect, but it is useful if you need to create a document quickly that will be viewable on a Windows system. 2. SCANNING We have a document feed flatbed scanner available. It can be used to convert papers into .pdf format. Instructions for using it are availabe here. It is attached to one of the general use Windows PCs, but priority should be given to people who need to use the scanner. Computing Services 1. EMAIL Our email server, mail.hydro.washington.edu is available for you to send and receive email. Your login (given when you arrived) is your email address. For example, to log in I use penglish. My email address is penglish@hydro.washington.edu To read and send email, log into the system mail.hydro.washington.edu otherwise known as dynamic.hydro.washington.edu then run "pine." Secure IMAP is also available (eg: for Netscape Mail), but is strongly recommended against due to the amount of trouble it has given us in the past. Pine is very simple and reliable by comparison.. Email can be used to transfer files, but it it is very inefficient method of doing so, and transferring large files could potentially disrupt the service for everyone. For this reason there is a 5 megabyte cap on email attachments. 2. FTP
For you To put a file on the ftp server, you can simply use SSH to securely copy it. Do: scp filename.txt ftp.hydro.washington.edu:/home/ftp/pub/yourusername This puts the file in your personal area. To put a file in a general area, so that it can be linked from the website and be persistant after you graduate, do scp filename.txt ftp.hydro.washington.edu:/home/ftp/pub/HYDRO/data/datasetname OR scp filename.txt ftp.hydro.washington.edu:/home/ftp/pub/HYDRO/model/modelname
To collect a file uploaded to the ftp server, copy it from the ftp incoming directory: scp ftp.hydro.washington.edu:/home/ftp/incoming/filename.txt . Note that this will require your collaborator to tell you what she called the file when it was uploaded.
For collaborators The files are uploaded to /home/ftp/incoming on ftp.hydro.washington.edu (currently ontario). You should be able to copy them from there, but due to security restrictions YOU CANNOT LIST THE DIRECTORY. This is why your collaborator must tell you the names of the files that they upload, per the instructions you send to them. When you are done, be sure to remove the files from the FTP server or request that Paul do it. 3. WWW We have a www site for providing information about the hydrology group and links to our datasets etc. We do not use the www site to provide data downloads because the space available on the www server is much more limited than on the ftp server. To create a professional web page as part of the group, copy someone else's web page, edit it and then tell the System admin and he will link it in for you, leaving you with the permissions to edit it in the future.
|