This is the second post of a series about how companies can transform into Cloud Native enterprises. To check the first post click here.


Understanding the Need for a Big Data Platform and High Impact Analytics

For companies who want to make their organizations more digital, installing new hardware and altering core systems architecture is not enough, developing a big data platform is key.

Big Data Platform

A big data platform is a complete stack of technology that enables three important functions: 1) collection, ingestion and storage of data from various databases, 2) synthesis and processing of data points using algorithms, 3) visualization of analytic output.

The benefit of creating this platform is it allows companies to unleash the power of the data they have been collecting over the years and may not be fully harnessing. When data is fed into a common platform and data points are explored, patterns could be uncovered to look at relationships over a period of time. From these patterns, new insights may surface, especially when putting variables that may not have been paired together before.

A common example is analyzing transactional (sales) data and media metrics. When overlaying variables such as media spend and sales, business leaders might see that a certain of range of media spend may equate to a certain level of sales. With observations like this, executives may draw the insight that media spend is a lever to pull when looking to increase sales toward a growth objective.

Another type of analysis that can be conducted involves customer-level data. This means that the data contains information to identify a transaction belonging to a specific customer. When a customer’s history of purchases and visits to a store is analyzed, companies will better understand their customers and identify which behaviors or groups are driving the highest revenue. This impacts marketing strategy and an outcome may be for the CMO to allocate his or her budget toward targeting customers who drive the highest revenue for the company.

Enterprise Data Lake

The concept of a data lake means putting various types and quantities of data into a platform that allows for the interaction of data points, resulting in breaking down of silos and enablement for deeper analysis.

This data platform not only acts as a storage for various datasets, but also as a conduit that enables organization to better mine their data and prepare for analytics projects.

Typical types of data that can be ingested into the data lake include: sales data, marketing data, and data from external sources. Having this system is one of the first steps on how a company can become more customer centric and data driven– by capturing and analysing customer data in one place– and uncover market opportunities.

This platform ingests data points that are synthesized using algorithms that are created in the data exploration layer. Additional tools are typically added to the data lake, which may include business analytics and other data mining capabilities.

Diagram 1: An Enterprise Data Lake ingests and integrates data from various sources and prepares it for exploration and analysis.

Data Exploration Layer

While it’s typical that historical data from an organization’s core systems are ingested first, data collected as part of the data lake may also include new data being collected, either as part of the data collection strategy of core systems, or new data sources are connected to the lake.

When data is collected on an ongoing basis and not limited to a specific initiative, algorithms are set in place in order to merge data points and explore trends in the data. The merging of old and new data could inspire new ways of looking at organizational challenges and enhance business strategies.

Data analysts typically go through a process that involve discovering, defining, designing and deploying analytics. Each phase is marked by the following activity:

  • Discover: Taking stock of current systems and available data sets.
  • Define: Creating the measurement strategy for analytics and insights generation.
  • Design: Designing the analytics project (methodology, reporting, data governance) and visualization requirements.
  • Deploy: Generating reports, interpreting results, and creating models to replicate analysis on other or larger data sets.

Visualization

In order for data to be meaningful for the organization, it cannot sit in storage, with its access only limited to the IT team. Unleashing the power of data has a lot to do with preparing the analytics output into charts and tables that can be accessed by various stakeholders. A typical visualization set up involves the use of modern dashboard tools such as Tableau or Qlikview, and it starts with the analytics team collaborating with business leads to do a design session, which involves understanding business objectives, KPIs, frequency of reporting required, and what kinds of charts and tables are most useful to be visualized. Analysts will recommend a dashboard design, which also includes an approach on how best to visualize data, given their familiarity with the organization’s existing data found in the data lake.

Diagram 2: After ingestion, developers and data architects create a Data Exploration Layer which allow analysts to explore the data. An enriched data stream is created that can be used by various applications and data visualization tools.

Conclusion

Organizations can now address challenges using the power of data and technology. New ways to putting together data points and analyzing them help uncover solutions, or even a new dimension to an organizational challenge.

In order to mobilize data throughout the organization and make it smarter, an enterprise could look into big data platforms. Key phases include data collection and storage, processing and computing, and producing an output that visualizes the data in a meaningful way in order to make insights useful immediately.  Ultimately, the goal is to leverage existing assets and empower the organization to make use of its own resource (proprietary data) in order to sustain and move itself forward into the digital economy, connecting with customers better.

DOWNLOAD: Legacy to Cloud Native Enterprise Architecture (PDF)

If you haven’t yet, you can download our complete Cloud Native Enterprise Architecture guide here.