Contents

Save small local files to forces

Save small local files to forces

It is important to save your results in a place that can last longer than a few days/weeks!
- When you have saved data locally on your JupyterLab instance and you want to mak a backup on https://forces2021.uiogeo-apps.sigma2.no/
import os
import pathlib
import s3fs
import xarray as xr

Get a sample file

ds = xr.tutorial.open_dataset("air_temperature.nc").rename({"air": "Tair"})

Save sample file into local file

  • The file is small (< 5GB so it is not an issue to do that)

ds.load().to_netcdf('Tair_temperature.nc')

<ipython-input-25-6f9ae4def414>:1: SerializationWarning: saving variable Tair with floating point data as an integer dtype without any _FillValue to use for NaNs
  ds.load().to_netcdf('Tair_temperature.nc')

Save your results to Remote private object storage

  • your credentials are in $HOME/.aws/credentials

  • check with your instructor to get the secret access key (replace XXX by the right key)

[default]
aws_access_key_id=forces2021-work
aws_secret_access_key=XXXXXXXXXXXX
aws_endpoint_url=https://forces2021.uiogeo-apps.sigma2.no/

target = s3fs.S3FileSystem(anon=False,
      client_kwargs={
         'endpoint_url': 'https://forces2021.uiogeo-apps.sigma2.no/'
      })

s3_prefix =  "s3://work/annefou/test/"
print(s3_prefix)

s3://work/annefou/test/

import glob
list_files = glob.glob("*.nc")

list_files

['Tair_temperature.nc']

for file in list_files:
    s3_path_file = os.path.join(s3_prefix, os.path.basename(file))
    print(file, s3_path_file)
    target.put(file, s3_path_file)

Tair_temperature.nc s3://work/annefou/test/Tair_temperature.nc

Then you can use the remote file

remote_file = ['work/annefou/test/Tair_temperature.nc']

fileset = [target.open(file) for file in remote_file]

ds_check = xr.open_mfdataset(fileset, combine='by_coords')
ds_check

<xarray.Dataset>
Dimensions:  (lat: 25, lon: 53, time: 2920)
Coordinates:
  * lat      (lat) float32 75.0 72.5 70.0 67.5 65.0 ... 25.0 22.5 20.0 17.5 15.0
  * lon      (lon) float32 200.0 202.5 205.0 207.5 ... 322.5 325.0 327.5 330.0
  * time     (time) datetime64[ns] 2013-01-01 ... 2014-12-31T18:00:00
Data variables:
    Tair     (time, lat, lon) float32 dask.array<chunksize=(2920, 25, 53), meta=np.ndarray>
Attributes:
    Conventions:  COARDS
    title:        4x daily NMC reanalysis (1948)
    description:  Data is from NMC initialized reanalysis\n(4x/day).  These a...
    platform:     Model
    references:   http://www.esrl.noaa.gov/psd/data/gridded/data.ncep.reanaly...

previous

Save files from bucket to bucket

next

Search and Load CMIP6 Data via ESGF / OPeNDAP