Direkt zum Inhalt | Direkt zur Navigation

Personal tools

Sektionen
You are here: Home » Data » Support Services on Data and Metadata » Data discovery » Script Based Download

Script Based Download

last modified Apr 24, 2019 10:16 AM

Attention: this page is currently under development - information will be updated soon.

Download several files together with scripts

 

Visit the ESGF user guide and ESGF FAQ to find detailed information for automatically generating Wget scripts or using your own Python scripts.

 

Wget scripts short tutorial

  • Find automatically generated scripts directly accessible from the ESGF portals
  • Wget scripts are UNIX Shell scripts that use Wget and run under Linux or Mac OSX
  • Fast access to collections: up to 1000 files can be simultaneously downloaded per script

 

How To

  • Open port 7512 (TCP) if you work behind a firewall
  • Login on in the portal hosting the data of your interest and add data to your DataCart
  • If you are not a member yet, register to your data access control group(s)
  • Download is limited to 1000 files per script. Optional but helpful: Reduce the number of files with the text filter function. Without the following sub-steps you will create a download script for whole datasets with all variables and that maybe many!
    • In "My Data Cart" fill in the names of the variables you need into the text field (step 1)
    • Click on "Apply" (step 2)

Text field for file selection in ESGF DataCart

  • Select "WGET Script"
  • You will get a browser notification to download a file called similar to "wget-some_long_number.sh". Press OK to download.
  • Optional but helpful: You may edit your script with a text editor. Delete lines with download files you don't need
  • Execute the script:
    • If you have Oracle Java 1.7+, use the default script call
bash wget-some_long_number.sh
    • Else try -H option
bash wget-some_long_number.sh -H
  • You will be asked for your OpenID and password. The script now contacts the portal where you are registered to retrieve security information
  • Your download now starts - take a coffee