{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "# AUX_OBS Ground Observatory Data\n", "\n", "> Abstract: We Demonstrate geomagnetic ground observatory data access through VirES - this is the AUX_OBS product distributed by BGS to support the Swarm mission, and contains data from [INTERMAGNET](http://intermagnet.github.io) and the [World Data Centre (WDC) for Geomagnetism](http://www.wdc.bgs.ac.uk). Data are available as three collections: 1 second and 1 minute cadences (INTERMAGNET definitive & quasi-definitive data), as well as specially derived hourly means over the past century (WDC).\n", "\n", "See also:\n", "- https://book.magneticearth.org/geomag-obs-models/01a_visualising-geomagnetic-observatory-data" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## About the data\n", "\n", "The data are also available from the BGS FTP server (`ftp.nerc-murchison.ac.uk/geomag/Swarm/AUX_OBS/`). If that is more useful to you, you can refer to [this older notebook demonstration](https://nbviewer.jupyter.org/urls/gist.githubusercontent.com/smithara/fd17d31fa4901fb9c663b81f12d4950c/raw/aaef54458f837c30a139b76903b43a50b7ae406d/04c1_Geomag-Ground-Data-FTP.ipynb) of access to the FTP server.\n", "\n", "Please note the data are under different usage terms than the Swarm data:\n", "- If you use the 1-second or 1-minute data, please refer to the [INTERMAGNET data conditions](https://intermagnet.github.io/data_conditions.html)\n", "- If you use the 1-hour data, please also refer to the [WDC usage rules](http://www.wdc.bgs.ac.uk/usage.html) and cite the article about the preparation of these data: \n", " [Macmillan, S., Olsen, N. Observatory data and the Swarm mission. Earth Planet Sp 65, 15 (2013). https://doi.org/10.5047/eps.2013.07.011](https://doi.org/10.5047/eps.2013.07.011)\n", "\n", "*Caution*: The magnetic vector components have been rotated into the geocentric (NEC) frame rather than the geodetic frame, so that they are consistent with the Swarm data. This is in contrast with the data provided directly from observatories." ] }, { "cell_type": "code", "execution_count": 1, "metadata": { "execution": { "iopub.execute_input": "2024-01-24T17:48:58.243952Z", "iopub.status.busy": "2024-01-24T17:48:58.243802Z", "iopub.status.idle": "2024-01-24T17:48:59.222775Z", "shell.execute_reply": "2024-01-24T17:48:59.222198Z" }, "tags": [] }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Python implementation: CPython\n", "Python version : 3.11.6\n", "IPython version : 8.18.0\n", "\n", "viresclient: 0.11.6\n", "pandas : 2.1.3\n", "xarray : 2023.12.0\n", "matplotlib : 3.8.2\n", "\n" ] } ], "source": [ "# Display important package versions used\n", "%load_ext watermark\n", "%watermark -i -v -p viresclient,pandas,xarray,matplotlib" ] }, { "cell_type": "code", "execution_count": 2, "metadata": { "execution": { "iopub.execute_input": "2024-01-24T17:48:59.224937Z", "iopub.status.busy": "2024-01-24T17:48:59.224419Z", "iopub.status.idle": "2024-01-24T17:48:59.461520Z", "shell.execute_reply": "2024-01-24T17:48:59.460949Z" } }, "outputs": [], "source": [ "from viresclient import SwarmRequest\n", "import matplotlib.pyplot as plt" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Organisation of data in VirES" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### AUX_OBS collection types\n", "\n", "Data are organised into AUX_OBSH (hour), AUX_OBSM (minute), AUX_OBSS (second) types. For example, to access the hourly data, use the *collection name* `SW_OPER_AUX_OBSH2_`." ] }, { "cell_type": "code", "execution_count": 3, "metadata": { "execution": { "iopub.execute_input": "2024-01-24T17:48:59.463759Z", "iopub.status.busy": "2024-01-24T17:48:59.463585Z", "iopub.status.idle": "2024-01-24T17:48:59.869590Z", "shell.execute_reply": "2024-01-24T17:48:59.869036Z" }, "tags": [] }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "{'AUX_OBSH': ['SW_OPER_AUX_OBSH2_']}\n", "{'AUX_OBSM': ['SW_OPER_AUX_OBSM2_']}\n", "{'AUX_OBSS': ['SW_OPER_AUX_OBSS2_']}\n" ] } ], "source": [ "request = SwarmRequest()\n", "print(request.available_collections(\"AUX_OBSH\", details=False))\n", "print(request.available_collections(\"AUX_OBSM\", details=False))\n", "print(request.available_collections(\"AUX_OBSS\", details=False))" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Within each collection, the following variables are available:" ] }, { "cell_type": "code", "execution_count": 4, "metadata": { "execution": { "iopub.execute_input": "2024-01-24T17:48:59.871425Z", "iopub.status.busy": "2024-01-24T17:48:59.871266Z", "iopub.status.idle": "2024-01-24T17:48:59.874486Z", "shell.execute_reply": "2024-01-24T17:48:59.874020Z" }, "tags": [] }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "['B_NEC', 'F', 'IAGA_code', 'Quality', 'ObsIndex']\n", "['B_NEC', 'F', 'IAGA_code', 'Quality']\n", "['B_NEC', 'F', 'IAGA_code', 'Quality']\n" ] } ], "source": [ "print(request.available_measurements(\"SW_OPER_AUX_OBSH2_\"))\n", "print(request.available_measurements(\"SW_OPER_AUX_OBSM2_\"))\n", "print(request.available_measurements(\"SW_OPER_AUX_OBSS2_\"))" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "- `B_NEC` and `F` are the magnetic field vector and intensity\n", "- `IAGA_code` gives the official three-letter IAGA codes that identify each observatory\n", "- `Quality` is either \"D\" or \"Q\" to indicate whether data is definitive (D) or quasi-definitive (Q)\n", "- `ObsIndex` is an increasing integer (0, 1, 2...) attached to the hourly data - this indicates a change in the observatory (e.g. of precise location) while the 3-letter IAGA code remained the same" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Data can be requested similarly to Swarm MAG products\n", "\n", "**Note that the `IAGA_code` variable is necessary in order to distinguish records from each observatory**\n", "\n", "**Note that there is a special message issued regarding the data terms**\n", "\n", "Let's fetch all the variables available within the 1-minute data, from two days:" ] }, { "cell_type": "code", "execution_count": 5, "metadata": { "execution": { "iopub.execute_input": "2024-01-24T17:48:59.876432Z", "iopub.status.busy": "2024-01-24T17:48:59.876092Z", "iopub.status.idle": "2024-01-24T17:49:04.534327Z", "shell.execute_reply": "2024-01-24T17:49:04.533811Z" }, "tags": [] }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "\n", "Accessing INTERMAGNET and/or WDC data\n", "Check usage terms at ftp://ftp.nerc-murchison.ac.uk/geomag/Swarm/AUX_OBS/minute/README\n", "\n" ] }, { "data": { "application/vnd.jupyter.widget-view+json": { "model_id": "f4ad75d8ddc44b9ea75e02ca6cc2d9fa", "version_major": 2, "version_minor": 0 }, "text/plain": [ "Processing: 0%| | [ Elapsed: 00:00, Remaining: ? ] [1/1] " ] }, "metadata": {}, "output_type": "display_data" }, { "data": { "application/vnd.jupyter.widget-view+json": { "model_id": "972d31f8d13c4ff4af8e517749d76aff", "version_major": 2, "version_minor": 0 }, "text/plain": [ "Downloading: 0%| | [ Elapsed: 00:00, Remaining: ? ] (20.375MB)" ] }, "metadata": {}, "output_type": "display_data" }, { "data": { "text/html": [ "
<xarray.Dataset>\n", "Dimensions: (Timestamp: 295200, NEC: 3)\n", "Coordinates:\n", " * Timestamp (Timestamp) datetime64[ns] 2016-01-01 ... 2016-01-02T23:59:00\n", " * NEC (NEC) <U1 'N' 'E' 'C'\n", "Data variables:\n", " Spacecraft (Timestamp) object '-' '-' '-' '-' '-' ... '-' '-' '-' '-' '-'\n", " Radius (Timestamp) float64 6.376e+06 6.36e+06 ... 6.367e+06 6.364e+06\n", " B_NEC (Timestamp, NEC) float64 3.804e+04 197.1 ... 764.8 4.623e+04\n", " Quality (Timestamp) <U1 'D' 'D' 'D' 'Q' 'D' 'D' ... 'D' 'D' 'Q' 'D' 'D'\n", " F (Timestamp) float64 4.297e+04 5.33e+04 ... 4.85e+04 4.962e+04\n", " Longitude (Timestamp) float64 72.87 18.82 171.8 58.6 ... 316.4 15.86 9.073\n", " Latitude (Timestamp) float64 18.5 68.23 -13.73 ... -22.26 47.74 53.56\n", " IAGA_code (Timestamp) <U3 'ABG' 'ABK' 'API' 'ARS' ... 'VSS' 'WIC' 'WNG'\n", "Attributes:\n", " Sources: ['SW_OPER_AUX_OBSM2__20160101T000000_20160101T235959_010...\n", " MagneticModels: []\n", " AppliedFilters: []
\n", " | F | \n", "Radius | \n", "Quality | \n", "Longitude | \n", "Latitude | \n", "IAGA_code | \n", "Spacecraft | \n", "B_NEC_N | \n", "B_NEC_E | \n", "B_NEC_C | \n", "
---|---|---|---|---|---|---|---|---|---|---|
Timestamp | \n", "\n", " | \n", " | \n", " | \n", " | \n", " | \n", " | \n", " | \n", " | \n", " | \n", " |
2016-01-01 00:00:00 | \n", "42974.178420 | \n", "6.375973e+06 | \n", "D | \n", "72.870 | \n", "18.503863 | \n", "ABG | \n", "- | \n", "38043.065649 | \n", "197.10 | \n", "19986.653510 | \n", "
2016-01-01 00:00:00 | \n", "53297.873523 | \n", "6.360062e+06 | \n", "D | \n", "18.823 | \n", "68.225754 | \n", "ABK | \n", "- | \n", "10737.698005 | \n", "1715.90 | \n", "52176.822927 | \n", "
2016-01-01 00:00:00 | \n", "38804.092366 | \n", "6.376929e+06 | \n", "D | \n", "171.781 | \n", "-13.726025 | \n", "API | \n", "- | \n", "32474.849888 | \n", "6910.60 | \n", "-20084.454605 | \n", "
2016-01-01 00:00:00 | \n", "56202.047989 | \n", "6.362618e+06 | \n", "Q | \n", "58.600 | \n", "59.231110 | \n", "ARS | \n", "- | \n", "15439.829840 | \n", "3848.14 | \n", "53902.445874 | \n", "
2016-01-01 00:00:00 | \n", "28516.997217 | \n", "6.377908e+06 | \n", "D | \n", "345.624 | \n", "-7.896461 | \n", "ASC | \n", "- | \n", "19779.633516 | \n", "-5398.80 | \n", "-19820.145984 | \n", "
... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "
2016-01-02 23:59:00 | \n", "53761.882434 | \n", "6.366376e+06 | \n", "D | \n", "236.583 | \n", "48.325950 | \n", "VIC | \n", "- | \n", "17905.402463 | \n", "5324.30 | \n", "50412.184986 | \n", "
2016-01-02 23:59:00 | \n", "38373.585718 | \n", "6.359150e+06 | \n", "D | \n", "351.718 | \n", "-70.562545 | \n", "VNA | \n", "- | \n", "18053.865463 | \n", "-4436.60 | \n", "-33569.429593 | \n", "
2016-01-02 23:59:00 | \n", "23289.026042 | \n", "6.375511e+06 | \n", "Q | \n", "316.350 | \n", "-22.264742 | \n", "VSS | \n", "- | \n", "16765.212732 | \n", "-6976.26 | \n", "-14582.118244 | \n", "
2016-01-02 23:59:00 | \n", "48503.461985 | \n", "6.367486e+06 | \n", "D | \n", "15.862 | \n", "47.736545 | \n", "WIC | \n", "- | \n", "20844.849807 | \n", "1427.50 | \n", "43772.597647 | \n", "
2016-01-02 23:59:00 | \n", "49624.347736 | \n", "6.364323e+06 | \n", "D | \n", "9.073 | \n", "53.559282 | \n", "WNG | \n", "- | \n", "18016.450576 | \n", "764.80 | \n", "46232.007071 | \n", "
295200 rows × 10 columns
\n", "" ], "text/plain": [ " F Radius Quality Longitude Latitude \\\n", "Timestamp \n", "2016-01-01 00:00:00 42974.178420 6.375973e+06 D 72.870 18.503863 \n", "2016-01-01 00:00:00 53297.873523 6.360062e+06 D 18.823 68.225754 \n", "2016-01-01 00:00:00 38804.092366 6.376929e+06 D 171.781 -13.726025 \n", "2016-01-01 00:00:00 56202.047989 6.362618e+06 Q 58.600 59.231110 \n", "2016-01-01 00:00:00 28516.997217 6.377908e+06 D 345.624 -7.896461 \n", "... ... ... ... ... ... \n", "2016-01-02 23:59:00 53761.882434 6.366376e+06 D 236.583 48.325950 \n", "2016-01-02 23:59:00 38373.585718 6.359150e+06 D 351.718 -70.562545 \n", "2016-01-02 23:59:00 23289.026042 6.375511e+06 Q 316.350 -22.264742 \n", "2016-01-02 23:59:00 48503.461985 6.367486e+06 D 15.862 47.736545 \n", "2016-01-02 23:59:00 49624.347736 6.364323e+06 D 9.073 53.559282 \n", "\n", " IAGA_code Spacecraft B_NEC_N B_NEC_E B_NEC_C \n", "Timestamp \n", "2016-01-01 00:00:00 ABG - 38043.065649 197.10 19986.653510 \n", "2016-01-01 00:00:00 ABK - 10737.698005 1715.90 52176.822927 \n", "2016-01-01 00:00:00 API - 32474.849888 6910.60 -20084.454605 \n", "2016-01-01 00:00:00 ARS - 15439.829840 3848.14 53902.445874 \n", "2016-01-01 00:00:00 ASC - 19779.633516 -5398.80 -19820.145984 \n", "... ... ... ... ... ... \n", "2016-01-02 23:59:00 VIC - 17905.402463 5324.30 50412.184986 \n", "2016-01-02 23:59:00 VNA - 18053.865463 -4436.60 -33569.429593 \n", "2016-01-02 23:59:00 VSS - 16765.212732 -6976.26 -14582.118244 \n", "2016-01-02 23:59:00 WIC - 20844.849807 1427.50 43772.597647 \n", "2016-01-02 23:59:00 WNG - 18016.450576 764.80 46232.007071 \n", "\n", "[295200 rows x 10 columns]" ] }, "execution_count": 6, "metadata": {}, "output_type": "execute_result" } ], "source": [ "df = data.as_dataframe(expand=True)\n", "df" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Use `available_observatories` to find possible IAGA codes\n", "\n", "We can get a dataframe containing the availability times of data from all the available observatories for a given collection:" ] }, { "cell_type": "code", "execution_count": 7, "metadata": { "execution": { "iopub.execute_input": "2024-01-24T17:49:05.215801Z", "iopub.status.busy": "2024-01-24T17:49:05.215639Z", "iopub.status.idle": "2024-01-24T17:49:11.884038Z", "shell.execute_reply": "2024-01-24T17:49:11.883479Z" } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "\n", "Accessing INTERMAGNET and/or WDC data\n", "Check usage terms at ftp://ftp.nerc-murchison.ac.uk/geomag/Swarm/AUX_OBS/minute/README\n", "\n" ] }, { "data": { "text/html": [ "\n", " | site | \n", "startTime | \n", "endTime | \n", "Latitude | \n", "Longitude | \n", "Radius | \n", "
---|---|---|---|---|---|---|
0 | \n", "AAA | \n", "2005-01-01T00:00:00+00:00 | \n", "2015-12-31T23:59:00+00:00 | \n", "NaN | \n", "NaN | \n", "NaN | \n", "
1 | \n", "AAE | \n", "1998-01-01T00:00:00+00:00 | \n", "2013-12-31T23:59:00+00:00 | \n", "NaN | \n", "NaN | \n", "NaN | \n", "
2 | \n", "ABG | \n", "1997-01-02T00:00:00+00:00 | \n", "2022-12-31T23:59:00Z | \n", "72.870 | \n", "18.503863 | \n", "6.375980e+06 | \n", "
3 | \n", "ABK | \n", "1997-01-02T00:00:00+00:00 | \n", "2023-12-25T23:59:00Z | \n", "18.800 | \n", "68.267960 | \n", "6.360051e+06 | \n", "
4 | \n", "AIA | \n", "2004-01-01T00:00:00+00:00 | \n", "2022-01-24T23:59:00Z | \n", "295.730 | \n", "-65.103358 | \n", "6.360536e+06 | \n", "
... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "... | \n", "
144 | \n", "VSS | \n", "1999-01-01T00:00:00+00:00 | \n", "2023-10-31T23:59:00Z | \n", "316.350 | \n", "-22.264742 | \n", "6.375511e+06 | \n", "
145 | \n", "WIC | \n", "2015-01-01T00:00:00+00:00 | \n", "2023-12-25T23:59:00Z | \n", "15.866 | \n", "47.736545 | \n", "6.367486e+06 | \n", "
146 | \n", "WNG | \n", "1997-01-02T00:00:00+00:00 | \n", "2023-11-30T23:59:00Z | \n", "9.053 | \n", "53.541246 | \n", "6.364345e+06 | \n", "
147 | \n", "YAK | \n", "2009-01-01T00:00:00+00:00 | \n", "2018-12-31T23:59:00Z | \n", "NaN | \n", "NaN | \n", "NaN | \n", "
148 | \n", "YKC | \n", "1997-01-02T00:00:00+00:00 | \n", "2020-01-24T23:59:00Z | \n", "245.518 | \n", "62.324005 | \n", "6.361545e+06 | \n", "
149 rows × 6 columns
\n", "<xarray.Dataset>\n", "Dimensions: (Timestamp: 2880, NEC: 3)\n", "Coordinates:\n", " * Timestamp (Timestamp) datetime64[ns] 2016-01-01 ... 2016-01-02T23:59:00\n", " * NEC (NEC) <U1 'N' 'E' 'C'\n", "Data variables:\n", " Spacecraft (Timestamp) object '-' '-' '-' '-' '-' ... '-' '-' '-' '-' '-'\n", " Radius (Timestamp) float64 6.36e+06 6.36e+06 ... 6.36e+06 6.36e+06\n", " B_NEC (Timestamp, NEC) float64 1.074e+04 1.716e+03 ... 5.198e+04\n", " Quality (Timestamp) <U1 'D' 'D' 'D' 'D' 'D' 'D' ... 'D' 'D' 'D' 'D' 'D'\n", " F (Timestamp) float64 5.33e+04 5.331e+04 ... 5.32e+04 5.32e+04\n", " Longitude (Timestamp) float64 18.82 18.82 18.82 ... 18.82 18.82 18.82\n", " Latitude (Timestamp) float64 68.23 68.23 68.23 ... 68.23 68.23 68.23\n", " IAGA_code (Timestamp) <U3 'ABK' 'ABK' 'ABK' 'ABK' ... 'ABK' 'ABK' 'ABK'\n", "Attributes:\n", " Sources: ['SW_OPER_AUX_OBSM2__20160101T000000_20160101T235959_010...\n", " MagneticModels: []\n", " AppliedFilters: []