{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "# Run Synsat on ICON Data for SEVIRI" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Here, we will \n", "- load ICON test data (limited area simulation, stored in separate files, from IFCES2 project)\n", "- run Synsat on ICON for SEVIRI and\n", "- finally store the Synsats into a netcdf file\n", "\n", "The workflow is similar to [02-Run-Synsat-for-ERA5-Data.ipynb](02-Run-Synsat-for-ERA5-Data.ipynb)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Setup Env and Load Libraries " ] }, { "cell_type": "code", "execution_count": null, "metadata": { "tags": [] }, "outputs": [], "source": [ "import os, sys\n", "os.environ['RTTOV_PYTHON_WRAPPER'] = '/work/bb1262/tools/rttov/rttov-v13.2/wrapper'\n", "\n", "import warnings\n", "warnings.filterwarnings(\"ignore\")" ] }, { "cell_type": "code", "execution_count": 2, "metadata": { "tags": [] }, "outputs": [], "source": [ "import synsatipy\n", "\n", "from synsatipy.synsat import SynSat\n", "\n", "import synsatipy.synsat_example_data as synsat_example_data" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Steps towards Synsat " ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Step I: Initialize Synsat class" ] }, { "cell_type": "code", "execution_count": 3, "metadata": { "tags": [] }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "... [synsat] set cloud / aerosol file to /work/bb1262/tools/rttov/rttov-v13.2/rtcoef_rttov13/cldaer_visir/sccldcoef_msg_3_seviri.dat\n", "... [synsat] load coefficient file /work/bb1262/tools/rttov/rttov-v13.2/rtcoef_rttov13/rttov13pred54L/rtcoef_msg_3_seviri_o3.dat\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "Load successful >>>>> inst_id : 1, nchannels : 6.\n", " 2025/05/07 08:20:14 Load coefficients:\n", " 2025/05/07 08:20:14 /work/bb1262/tools/rttov/rttov-v13.2/rtcoef_rttov13/rttov13pred54L/rtcoef_msg_3_seviri_o3.dat\n", " 2025/05/07 08:20:14 /work/bb1262/tools/rttov/rttov-v13.2/rtcoef_rttov13/cldaer_visir/sccldcoef_msg_3_seviri.dat\n" ] } ], "source": [ "s = SynSat( synsat_instrument = 'SEVIRI' )" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Step II: Load Example Data " ] }, { "cell_type": "code", "execution_count": 4, "metadata": { "tags": [] }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "... [synsat] read data from file /work/bb1376/data/icon/atlantic-cases/paulette/ifces2-atlanXL-20200907-exp021/POSTPROC//3d_full_base_DOM02_ML_20200912T000000Z_regrid7km.nc\n" ] } ], "source": [ "iconname = synsat_example_data.get_example_data( 'icon02' )\n", "s.load( iconname )" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "The ICON data have been loaded into a data handler. You can inspect the data as follows:" ] }, { "cell_type": "code", "execution_count": 5, "metadata": { "tags": [] }, "outputs": [], "source": [ "icon = s.synsat.data_handler.input_data" ] }, { "cell_type": "code", "execution_count": 6, "metadata": { "tags": [] }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ " Size: 2GB\n", "Dimensions: (lon: 1094, lat: 767, lev: 70, time: 1)\n", "Coordinates:\n", " * lon (lon) float32 4kB -85.0 -84.94 -84.87 ... -15.18 -15.11 -15.05\n", " * lat (lat) float32 3kB 0.0 0.06 0.12 0.18 ... 45.78 45.84 45.9 45.96\n", " * height (lev) float64 560B 1.0 2.0 3.0 4.0 5.0 ... 66.0 67.0 68.0 69.0 70.0\n", " * time (time) datetime64[ns] 8B 2020-09-12\n", "Dimensions without coordinates: lev\n", "Data variables:\n", " p (time, lev, lat, lon) float32 235MB dask.array\n", " t (time, lev, lat, lon) float32 235MB dask.array\n", " q (time, lev, lat, lon) float32 235MB dask.array\n", " clwc (time, lev, lat, lon) float32 235MB dask.array\n", " ciwc (time, lev, lat, lon) float32 235MB dask.array\n", " cswc (time, lev, lat, lon) float32 235MB dask.array\n", " SKT (time, lat, lon) float32 3MB dask.array\n", " T2M (time, lat, lon) float32 3MB dask.array\n", " SP (time, lat, lon) float32 3MB dask.array\n", " cc (time, lev, lat, lon) float32 235MB dask.array\n" ] } ], "source": [ "print(icon)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "The actual ICON data are not really small. At this time, however, the data are not loaded into memory. \n", "\n", "Maybe this is really too large for our small example here. **Let's redo the input and subset the ICON data!**\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Step IIb: Subset and Thean Load Again Example Data " ] }, { "cell_type": "code", "execution_count": 7, "metadata": { "tags": [] }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "... [synsat] read data from file /work/bb1376/data/icon/atlantic-cases/paulette/ifces2-atlanXL-20200907-exp021/POSTPROC//3d_full_base_DOM01_ML_20200912T000000Z_regrid7km.nc\n" ] } ], "source": [ "iconname = synsat_example_data.get_example_data( 'icon01' )\n", "s.load( iconname, isel = {'lon': slice(0,None,4), 'lat': slice(0,None, 4)} )" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "The ICON data have been loaded into a data handler. You can inspect the data as follows:" ] }, { "cell_type": "code", "execution_count": 8, "metadata": { "tags": [] }, "outputs": [], "source": [ "icon = s.synsat.data_handler.input_data" ] }, { "cell_type": "code", "execution_count": 9, "metadata": { "tags": [] }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ " Size: 104MB\n", "Dimensions: (lon: 274, lat: 192, lev: 70, time: 1)\n", "Coordinates:\n", " * lon (lon) float32 1kB -85.0 -84.74 -84.49 ... -15.62 -15.37 -15.11\n", " * lat (lat) float32 768B 0.0 0.24 0.48 0.72 ... 45.12 45.36 45.6 45.84\n", " * height (lev) float64 560B 1.0 2.0 3.0 4.0 5.0 ... 66.0 67.0 68.0 69.0 70.0\n", " * time (time) datetime64[ns] 8B 2020-09-12\n", "Dimensions without coordinates: lev\n", "Data variables:\n", " p (time, lev, lat, lon) float32 15MB dask.array\n", " t (time, lev, lat, lon) float32 15MB dask.array\n", " q (time, lev, lat, lon) float32 15MB dask.array\n", " clwc (time, lev, lat, lon) float32 15MB dask.array\n", " ciwc (time, lev, lat, lon) float32 15MB dask.array\n", " cswc (time, lev, lat, lon) float32 15MB dask.array\n", " SKT (time, lat, lon) float32 210kB dask.array\n", " T2M (time, lat, lon) float32 210kB dask.array\n", " SP (time, lat, lon) float32 210kB dask.array\n", " cc (time, lev, lat, lon) float32 15MB dask.array\n" ] } ], "source": [ "print(icon)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Yes, much better and smaller now!" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Step III: Setup Run Options and Start Execution " ] }, { "cell_type": "code", "execution_count": 10, "metadata": { "tags": [] }, "outputs": [], "source": [ "s._options.Nthreads=4\n", "s._options.NprofsPerCall = 4000" ] }, { "cell_type": "code", "execution_count": 11, "metadata": { "tags": [] }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "... [synsat] running 0/14 chunk with {'profile': slice(0, 4000, None)}\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "IR emissivity atlas loaded successfully\n", "Atlas deallocated.\n", " 2025/05/07 08:20:24 Running RTTOV using nthreads = 4 and nprofs_per_call = 4000\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "... [synsat] running 1/14 chunk with {'profile': slice(4000, 8000, None)}\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "IR emissivity atlas loaded successfully\n", "Atlas deallocated.\n", " 2025/05/07 08:20:28 Running RTTOV using nthreads = 4 and nprofs_per_call = 4000\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "... [synsat] running 2/14 chunk with {'profile': slice(8000, 12000, None)}\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "IR emissivity atlas loaded successfully\n", "Atlas deallocated.\n", " 2025/05/07 08:20:34 Running RTTOV using nthreads = 4 and nprofs_per_call = 4000\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "... [synsat] running 3/14 chunk with {'profile': slice(12000, 16000, None)}\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "IR emissivity atlas loaded successfully\n", "Atlas deallocated.\n", " 2025/05/07 08:20:38 Running RTTOV using nthreads = 4 and nprofs_per_call = 4000\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "... [synsat] running 4/14 chunk with {'profile': slice(16000, 20000, None)}\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "IR emissivity atlas loaded successfully\n", "Atlas deallocated.\n", " 2025/05/07 08:20:42 Running RTTOV using nthreads = 4 and nprofs_per_call = 4000\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "... [synsat] running 5/14 chunk with {'profile': slice(20000, 24000, None)}\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "IR emissivity atlas loaded successfully\n", "Atlas deallocated.\n", " 2025/05/07 08:20:46 Running RTTOV using nthreads = 4 and nprofs_per_call = 4000\n", " 2025/05/07 08:20:47 rttov_check_reg_limits.F90\n", " Input water vapour profile exceeds upper coef limit (profile number = 989)\n", " 2025/05/07 08:20:47 Limit = 20.3040\n", " 2025/05/07 08:20:47 p (hPa) = 97.1505\n", " 2025/05/07 08:20:47 Value = 20.7966\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "... [synsat] running 6/14 chunk with {'profile': slice(24000, 28000, None)}\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "IR emissivity atlas loaded successfully\n", "Atlas deallocated.\n", " 2025/05/07 08:20:51 Running RTTOV using nthreads = 4 and nprofs_per_call = 4000\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "... [synsat] running 7/14 chunk with {'profile': slice(28000, 32000, None)}\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "IR emissivity atlas loaded successfully\n", "Atlas deallocated.\n", " 2025/05/07 08:20:56 Running RTTOV using nthreads = 4 and nprofs_per_call = 4000\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "... [synsat] running 8/14 chunk with {'profile': slice(32000, 36000, None)}\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "IR emissivity atlas loaded successfully\n", "Atlas deallocated.\n", " 2025/05/07 08:21:02 Running RTTOV using nthreads = 4 and nprofs_per_call = 4000\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "... [synsat] running 9/14 chunk with {'profile': slice(36000, 40000, None)}\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "IR emissivity atlas loaded successfully\n", "Atlas deallocated.\n", " 2025/05/07 08:21:06 Running RTTOV using nthreads = 4 and nprofs_per_call = 4000\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "... [synsat] running 10/14 chunk with {'profile': slice(40000, 44000, None)}\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "IR emissivity atlas loaded successfully\n", "Atlas deallocated.\n", " 2025/05/07 08:21:11 Running RTTOV using nthreads = 4 and nprofs_per_call = 4000\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "... [synsat] running 11/14 chunk with {'profile': slice(44000, 48000, None)}\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "IR emissivity atlas loaded successfully\n", "Atlas deallocated.\n", " 2025/05/07 08:21:16 Running RTTOV using nthreads = 4 and nprofs_per_call = 4000\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "... [synsat] running 12/14 chunk with {'profile': slice(48000, 52000, None)}\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "IR emissivity atlas loaded successfully\n", "Atlas deallocated.\n", " 2025/05/07 08:21:20 Running RTTOV using nthreads = 4 and nprofs_per_call = 4000\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "... [synsat] running 13/14 chunk with {'profile': slice(52000, None, None)}\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "IR emissivity atlas loaded successfully\n", "Atlas deallocated.\n", " 2025/05/07 08:21:24 Running RTTOV using nthreads = 4 and nprofs_per_call = 4000\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "CPU times: user 1min 13s, sys: 52.6 s, total: 2min 5s\n", "Wall time: 1min 6s\n" ] } ], "source": [ "%%time\n", "\n", "s.run( chunked = True )" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Step IV: Extract the Output" ] }, { "cell_type": "code", "execution_count": 12, "metadata": { "tags": [] }, "outputs": [], "source": [ "icon_synsat = s.extract_output()" ] }, { "cell_type": "code", "execution_count": 13, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ " Size: 3MB\n", "Dimensions: (time: 1, lon: 274, lat: 192)\n", "Coordinates:\n", " * time (time) datetime64[ns] 8B 2020-09-12\n", " * lon (lon) float32 1kB -85.0 -84.74 -84.49 ... -15.62 -15.37 -15.11\n", " * lat (lat) float32 768B 0.0 0.24 0.48 0.72 ... 45.12 45.36 45.6 45.84\n", "Data variables:\n", " bt062 (time, lon, lat) float64 421kB 226.2 226.2 226.1 ... 233.6 232.2\n", " bt073 (time, lon, lat) float64 421kB 244.4 244.3 244.2 ... 257.2 256.5\n", " bt087 (time, lon, lat) float64 421kB 278.9 278.9 278.8 ... 285.9 286.0\n", " bt108 (time, lon, lat) float64 421kB 282.8 282.7 282.6 ... 287.9 288.1\n", " bt120 (time, lon, lat) float64 421kB 280.9 280.8 280.8 ... 287.1 287.4\n", " bt134 (time, lon, lat) float64 421kB 252.2 252.1 252.1 ... 264.1 264.3\n", "Attributes:\n", " author: Fabian Senf\n", " contact: senf@tropos.de\n", " institution: Leibniz Institute for Tropospheric Research\n", " creation_time: 2025-05-07 08:21:24.578753\n", " synsat_version: 0.1\n", " synsat_githash: 4f8b856c0aceda7924506f3c05b728f26ad0190a\n", " license: CC-BY SA 3.0\n", " _local_software_path: /home/b/b380352/proj/2022-01_RTTOV-Synsat/synsatip...\n", " input_filename: /work/bb1376/data/icon/atlantic-cases/paulette/ifc...\n" ] } ], "source": [ "print( icon_synsat )" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Step V: Store Data " ] }, { "cell_type": "code", "execution_count": 14, "metadata": { "tags": [] }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "... store synsat at ./Data/seviri_synsat_icon_example_data.nc\n" ] } ], "source": [ "outdir = './Data'\n", "\n", "if not os.path.isdir( outdir ):\n", " os.makedirs( outdir )\n", "outfile = f'{outdir}/seviri_synsat_icon_example_data.nc'\n", "\n", "print(f'... store synsat at {outfile}')\n", "icon_synsat.to_netcdf(outfile)\n" ] } ], "metadata": { "kernelspec": { "display_name": "SynSatiPy (Python 3.10)", "language": "python", "name": "python3-synsatipy" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.10.10" } }, "nbformat": 4, "nbformat_minor": 4 }