2019 will mark the 20-year anniversary of the XMM-Newton Mission. So far, the mission has successfully completed a total of around 14.000 pointing observations, and it is expected to continue for many more years, producing a huge number of high-quality science data products. Data processing of those observations is carried out by the XMM-Newton Pipeline Processing System (PPS) and the products are delivered to the XMM-Newton Science Archive (XSA). During the two decades many changes have been implemented in the data processing software, partly following improvements to the calibration of the science instruments. Several re-processing campaigns have been undertaken along the mission in order to have an up-to-date and uniformly processed set of high-level science data products in the archive. A new re-processing exercise is to be carried out, as it has been more than six years since the last re-processing campaign in 2011. Unlike the daily mission operations where a a limited number of observations have to be processed by PPS, a whole mission re-processing is a real challenge. An individual XMM-Newton Pipeline job (of one observation) can take up to five hours of computer processing time, some of them even longer. To achieve the processing of thousands of observations in a reasonable period of time requires a special preparation including a deep analysis of the computing resources. An extreme optimization of the resources sharing becomes essential in our case. Besides the optimization of the computing infrastructure usage, a set of software tools had to be developed in order to cope with the management and monitoring of this enormous number of individual Pipeline jobs.
Link to PDF (may not be available yet): P4-14.pdf