So today I started a little project to look in to how the amount activity performed by NHS providers changes over time.  The focus is going to be on the aggregate amount of activity performed by each individual provider.  There are potentially a number of stories in these data but the most interesting is whether it is possible to forecast the amount of activity for the next year.

Today I have started building my dataset from the annual activity datasets published by the Health and Social Care Information Centre (HSCIC).  I was hoping that the collection of the data into a pleasing balanced panel would be smooth going.  How wrong I was.

Firstly, one would have thought that in this age of Data.gov.uk  one of the largest holders of publicly accessible data in the UK would have a sophisticated system for storing and searching through their data.  The huge value of the data that has been fully uploaded to Data.gov.uk is that it is very easy to find different years etc. of the same data, or different categories.  Like the Office for National Statistics, the HSCIC has chosen to take its time in systematising the storage of its data.  It is possible to search for the Hospital Episode Statistics on Data.gov.uk, but you will not find any spreadsheets, let alone tidy csv files.  All you will find is links to HSCIC pages.  Welcome to the 90s!

Once you’ve resigned yourself to trawling through the HSCIC pages you will encounter multiple frustrations with corralling together time-series data: each year’s publication has it’s own page with inconsistent titling so they don’t all come up together in a search; the spreadsheet for each year’s data has a different structure so you can’t pull it out with a script and the same variables have different names in different years.  It’s almost like someone is intentionally trying to make it hard to do anything with this data…

For anyone reading this with knowledge of these things, I have only been working on data for Admitted Patient Care (APC).  The other big category of hospital activity is Outpatient Procedures (OpProc).  The major difference for these purposes is that the activity measured is different.  For APC, the metric is Finished Consultant Episodes,   whereas in OpProc measures procedures.  For any sort of inferential model of hospital activity it will be necessary to look at both because for some conditions a patient can be admitted or treated as an outpatient.

More to follow as I stick my head further under the hood!