Written by Audrey Pilcher

Small organizations, enterprises, and even governments are struggling to keep up with demand for actionable insights from meaningful information. To enable data-driven policy and practice, they use predictive analytics and machine learning, but the scope of their use is significantly limited.

But why leading public service agencies and even large commercial organizations cannot apply technologies like predictive analytics and machine learning to the full? Are their computational models not mature enough?

Well, the answer lies in the readiness of Big Data platforms and the ability to interrogate immense data sets in real-time.

The Need for Insights

Do decisions makers have sufficient information to make informed choices and monitor the effectiveness of their policies and programs? As it turns out, yes, because the amount of data people and organizations produce every day is mind-boggling.

According to Domo, there are 2.5 quintillion bytes of data created every day by social networks, public services like Uber, Google, and the Weather Channel. With the number of Internet users growing and new technology like the Internet of Things developing, the pace of data production will only accelerate.

While it’s already pretty hard to wrap your mind around the amount of data we’re producing, the upcoming years will be even more impressive. For companies and government agencies, this means one thing: more data that needs to be analyzed as soon as possible.

Naturally, analyzing such massive amounts of data is extremely hard, to say the least. However, this is something they absolutely have to do. For example, in a study of 502 executives conducted by InterSystems, 75 percent of the respondents thought that untimely data inhibited business opportunities.

Moreover, 27 percent of the sample also reported that untimely data negatively affected productivity/agility of their companies while 54 percent also claimed that it limited operational efficiency.

Similar urgency is felt in governments that need data to conduct evidence-based policymaking. For example, a recent report by the Commission on Evidence-Based Policymaking highlighted the importance of data-driven government. According to the report, one of the essential requirements for a government that makes effective decisions and policies is the availability of good information.

Here’s how the authors explain this requirement:

“The American people want a government that solves problems. This requires that decision makers have good information to guide their choices about how current programs and policies are working and how they can be improved.” – Commission on Evidence-Based Policymaking.

Achieving such critical government functions like enhancing economic development, collecting taxes, and determining eligibility for benefits requires a considerable amount of information. To ensure that the government performs these and other functions effectively, the Commission recommends using the strategy of evidence-based policymaking, which involves the use of already collected data and using sophisticated analysis tools to unlock insights for addressing the country’s biggest challenges.

Commission on Evidence-Based Policymaking sees technology as the answer to enabling data-driven policy and effective data analysis. Specifically, the organization advises to make the use of sophisticated technology a part of routine government operations and make it one of the deciding factors defining effective public policy.

How to Overcome Big Data Challenges

The term “big data” refers to massive amounts of structured and unstructured data that organizations, governments, and other decision makers can mine and analyze for business gains, public policy creation, and other purposes.

To overcome challenges associated with such high-volume, high-velocity data, organizations can use real-time computing strategy. Real-time computing is a system that responds to changes according to pre-developed time constraints, usually milliseconds.

This branch of computer science is actually not a new phenomenon; in fact, there are scientific papers on real-time computing dating back to 1994. However, with a rapid advancement of technology, it became highly relevant today because of the rapidly increasing volumes of data that require analysis.

Two strategies that governments and commercial organizations can employ to overcome big data challenges and obtain valuable insights are centralized data governance model or near-real-time analytical technology.

“A centralized data governance model with decentralized execution involves a large, centralized body determining the framework of controls and stakeholders creating their personalized parts of master data,” explains Simon Grinspoon, a data analyst at A-Writer.

The State Of Indiana’s Management Performance Hub (MPH) is the best example of using real-time computing and other sophisticated technologies in governments. It delivers data-driven decision making, data-informed policymaking, and provides analytics solutions tailored to public management.

Another way to overcome big data challenges is to take advantage of near-real-time analytics solutions across distributed data platforms. According to Intel’s Big Data Technologies for Near-Real-Time Results whitepaper, these solutions are rapidly evolving and include Hadoop (A framework for generating value from Big Data) and NoSQL databases such as Apache Cassandra, HBase, and MongoDB that handle massive amounts of structured and unstructured data.


Big Data analytics technology such as real-time computing has significant benefits for governments and commercial organizations because it allows the storage and analysis of massive amounts of data. Ultimately, technology will be used to ensure evidence-based policymaking, which takes data governance and quality of public policy creation to a whole new level.


Audrey is a passionate blogger and marketer. Her areas of interests are very wide, but mostly she writes about content marketing and innovative business technologies.  Her aim is to engage people to self-growth and staying motivated.

Pin It on Pinterest

Share This

Share this post with your friends!