During the 2000’s I worked on a project with Leicester University’s Radio and Space Plasma Physics group that was studying the effects that space weather had on the modification of the Earth’s atmospheric air mass circulation system.
The project called forth a number of different aspects from my expertise including Statistics, Mathematics and Computing. We had to deal with very large (at that time) climate datasets and the analysis work was undertaken using the latest super-computing resources at our disposal.
My part on the project was the careful selection of theoretically sound statistical methodologies that we could deploy in order to advance the science beyond what was being done at the time. Much to my surprise, after researching the methods used by other scientists, their statistical methodology was, how can I say, rudimentary and non-robust, and perhaps even inappropriate as far as the data was concerned. It was my task to ensure that we were using the very best methods for the data at hand. Climate data is full of non-linearities that simply violate the assumptions that classical linear statistical analysis rely on.
Our research and results were quite literally ground breaking, and we successfully pushed the boundaries into regions that had not been achieved previously by anyone else. It was a great privilege for me to work on such a project. It certainly pushed me professionally and at times potentially put my own neck on the line so to speak. I really had to “get it right”! When one works on a project of this complexity, normal, more every day data analysis is so much easier!
I had some of my research published in the The Philosophical Transactions of The Royal Society, not a paper authored by myself, but still very nice to see my work reproduced in such a prestigious scientific journal.
As our research progressed the principle project scientist was unhappy with the datasets that other scientists were using to study space weather and climate interactions. Things needed to be stepped up a gear. He stated that rather than using temperature and wind flow data, that other scientists seemed to have gotten stuck with, we needed to study space weather in conjunction with Potential Vorticity.
The correlation between Ap and Potential vorticity for the years 1957 to 2001 (daily resolution data). Ap is the index of disturbance in the Geomagnetic field due to Solar forcing in terms of energetic particles and radiation originating mostly from the Sun.
This plot demonstrates the impact of the geomagnetic field on the North Polar Vortex as exemplified by the very statistically significant positive correlations in the North pole region. The North polar vortex plays a significant role in determining climate and meteorological patterns in the Northern hemisphere.
Potential Vorticity, in the climate community, is regarded as a primary atmospheric circulation tracer. If we could demonstrate that space weather events were associated with changes in Potential Vorticity, then we would certainly be on to something of importance as far as space weather being a contributing factor to climate modification.
The big problem was there were no available Potential Vorticity datasets that we could use. The few organisations at that time (who we approached), would not let us get our hands on them! The only option was for us to generate our own dataset.
It fell to myself to generate our own in-house Potential Vorticity dataset. At the time I thought “Well OK, I’ll do it”, thinking that the task would perhaps take only days to complete. I was soon advised that the computation of Potential Vorticity was not as simple as that! A sub-project was commissioned so that I could grapple with the problem over a several month period.
The correlation between F10.7 and Air temperature for the years 1957 to 2001 (daily resolution data). F10.7 is the standard index used to measure solar activity and is the 10.7cm radio flux energy emitted by the Sun.
Similar to the above plot, this one demonstrates how Solar activity impacts air temperature in the North polar region.
The maths was hard enough, and it had to be all implemented numerically. I utilised the method of computing Potential Vorticity from wind circulation and temperature data. The input data was so noisy that I had to employ judicious amounts of signal processing in order to prevent the numerical analysis algorithms that I was developing from “blowing up” in my face.
About nine months later, and sprouting some new grey hairs, I solved the entire problem. We spot checked my freshly generated dataset with an array of samples by those held by external organisations and my data fitted with what they had. They even checked my data with their own jealously guarded datasets! All in all it was a successful outcome. I even had the accolade bestowed upon me that I was only one of three people on the planet who was able to compute Potential Vorticity, which one of those shock-horror moments of my life! I expect everybody can do it now, probably on their wrist watches.
It is all such a long time ago, and I feel comfortable talking about it now. It seems like it was “that other Chris” who did it all anyway so it does not really even matter! I expect the truth can finally be told.