Big data technology is beginning to make a mark within the UK public sector.
Both HM Revenue and Customs (HMRC) and the Home Office are now using Hadoop to help manage their data needs, according to a report from Computer Weekly. Both organisations are using commercial distributors of the software, with HMRC utilising Cloudera and the Home Office working with Hortonworks.
The former spent some £7.4 billion on Cloudera at the beginning of last year before investing almost another £1 million earlier in 2016.
A HMRC spokesperson told Computer Weekly: "HMRC has built an enterprise data hub – a powerful central repository for all of its data, which will help it to personalise services to customers and strengthen its compliance work. HMRC will be able to store and analyse data using a mix of open source and closed source tools, and commodity hardware, representing better value for money for taxpayers."
The spokesperson added that use of Hadoop will allow for greater operational efficiency and a level of analytical capacity that has not been available to it in the past.
Records show the Home Office spent £53,000 with Hortonworks in August 2014 and a further £61,000 two months later. It is reported that the organisation is using Hadoop to connect the various databases it currently relies on.
Regarding the use of big data technology in the public sector, Cloudera's vice-president for northern Europe Stephen Line said full-scale adopting will take time.
"Government, like a lot of old industry, has to go through that digital transformation, modernising its digital architecture, breaking down those silos. The UK is not necessarily behind or ahead particularly," he commented.
Other UK public sector bodies now utilising big data technology include the National Crime Agency and Office for National Statistics.
Earlier this year, an independent report commissioned by the UK government called for the nation's public sector to improve its use of data. Professor Sir Charlie Bean, a former deputy governor of the Bank of England, compiled the document, which said improvements need to be made to ensure that accurate economic statistics on the digital economy can be captured.
Prof Bean: "We need to be candid about the limitations of UK economic statistics. The UK was one of the original pioneers of national accounting. We need to take economic statistics back to the future or we risk missing out an important part of the modern economy from official figures."
Among his recommendations were for the establishment of two new centres to better measure the UK economy and unlock the "treasure trove of big data available – especially in the public sector".
Prof Bean called on the Office for National Statistics to become innovative enough to provide the kind of data the country needs.