Leighton Gosnell, Head of Data, Merkle Aotearoa.
Today we start the series by looking at what we need to do and the principles we need to follow, to get our data foundations right – and, of course, why this matters.
Anyone who has spent any time with me talking about the ins and outs of becoming data-driven knows I love a good let’s imagine we’re building a house… analogy.
It’s an effective and simple way to provide context and make things more concrete (excuse the pun). Because, while most people struggle with the ‘why’ of data architecture, which is abstract and conceptual by nature, everyone has at some stage seen a building going up and can understand, to varying degrees, the parts involved in the process.
So, let’s begin.
GETTING YOUR BLUEPRINT RIGHT
Data architecture is like the blueprint for a building, which outlines the design and layout, including the location of rooms, walls and infrastructure. Data architecture outlines the design and layout of a data system, including the location of data sources, data storage, and data processing.
Where a blueprint must consider the size and function of the building, the number of occupants and the intended use of the space, data architecture must consider the size and complexity of the data system, the number of users and the intended use of the data.
Building blueprints need to be well thought-through, with flexibility and consideration to accommodate changes and future expansion. Likewise, data architecture needs to be able to accommodate changes and growth in data volume, velocity, and variety.
THE CORE INGREDIENTS FOR GETTING YOUR 'BLUEPRINT' RIGHT
The foundation of modern data architecture is built on several key principles which, when applied correctly, will help your organisation to unlock the full value of its data.
1. Target business challenges and opportunities first
First things first, do you have a clear understanding of the business problem/opportunity that your data architecture/blueprint is looking to solve?
Do you know what your ‘house’ needs to do (and why), who it will accommodate and how it might need to change in time to adapt to its residents’ future needs?
Have you identified a business sponsor who can clearly articulate and support a strong and viable business case for the build and the requirements for your blueprint?
Don’t focus on the technology - it is a means to an end, not an end goal in itself.
A modern data architecture needs to be tailored to the specific needs of your organisation and the problem you are trying to solve. A good way to ensure you really understand this is to take an agile and iterative approach to design (for example by using design sprints), and work closely with your business users to focus on the business problem and opportunity at hand and refine your data strategy accordingly.
Only once your business problem/opportunity and success criteria have been clearly defined should you then choose the tools that best fit the job and support your data architecture.
2. Make sure your data is accessible
Just as a building needs easy access to all its rooms and spaces for the inhabitants to efficiently move around and accomplish their tasks, your data platform should have easy access to all data for your business to operate smoothly and your teams to access the data they need with minimal restrictions.
Being able to efficiently access and utilise data is crucial for solving business problems. It allows organisations to make data-driven decisions, identify patterns and trends and gain insights that would otherwise be difficult to uncover. By utilising data effectively, your organisation can better understand and respond to the needs of your customers and your employees, improve operations, identify new opportunities and make better predictions about the future.
3. Design with flexibility in mind, right from the start
A versatile data architecture is essential for dealing with a wide range of data formats (such as structured, semi-structured and unstructured).
These days it’s rare for organisations to implement a single monolithic data environment enabling the storage and processing of every business need. The more common approach is to design a contemporary data architecture that is likely to contain multiple data solutions and storage environments.
Data lakes are often used for big data analytics and machine learning projects, where the focus is on storing and processing large amounts of unstructured data at low cost. Data warehouses, on the other hand, are more commonly used for business intelligence and reporting, where the focus is on providing fast and efficient access to structured data for decision-making.
Knowing the right combination of tools at the outset will allow your business to leverage and grow and will prevent unnecessary re-work down the track.
The organisation that has to rebuild the least over time will win the ‘race’, every time. Data isn’t simple and ultimately it costs a lot more to get it wrong than it does to take the time, upfront, to get it right.
A system built with scalability, maintainability and flexibility in mind will allow your organisation to focus on adding new features and capabilities instead of constantly fixing or redesigning the existing system. This will ultimately lead to a more efficient and cost-effective development process, allowing your business to ‘win the race’ when it comes to delivering value to your customers and staying competitive in the market.
4. Go all-in with cost-efficient cloud capabilities
‘Cloud-native’ data architecture (optimised for deployment and operation in a cloud computing environment) allows your organisation to take advantage of the scalability and cost-efficiency offered by cloud-based infrastructure and services, such as serverless and managed services.
Serverless computing, for example, is a popular cloud-native option, allowing for the automatic scaling of resources, which means businesses only pay for the compute and storage resources they use.
Given many cloud-based tools are pay-as-you go and you’re only billed for the time you spend actively using them, your organisation can scale the compute resources you need while keeping storage costs low by only keeping the data you need. And by scaling the resources you need, as you need them, without over-provisioning, you’ll set your organisation up for cost savings and better performance.
5. Scale for success
As your organisation grows, so too, will the demand on your data architecture. This can manifest in a variety of ways, such as increased storage and processing needs, and can change depending on the season, day of the week, or even the time of day.
To keep up with this demand, having a scalable solution is key. It's tempting to simply buy more hardware to increase capacity. However, this approach is risky, as it's difficult to predict exactly how much capacity you'll need at any given time. Over-provisioning resources can be costly, and under-provisioning can lead to missed insights and lost revenue.
Scalability is key to addressing this challenge. Whether it's scaling up to handle more resource-intensive queries, or scaling out to handle more queries at once, the ability to adapt to changing demand is essential.
Cloud-based, scalable architectures are an excellent solution, allowing you to adjust your resources as needed and avoid the risks of overspending or under-serving.
There are a number of cloud-based data architecture solutions out there, so it’s critical to understand the future benefits of each solution. Make sure you get independent advice to clarify and validate that what’s on offer can be delivered in the manner that works best for your business and will achieve the outcomes that deliver upon the experience you want your customers and your organisation to have.
IN CONCLUSION
With the right data architecture, principles and business outcome-focused data strategy established, you’ll have well-laid foundations that will set you up for success. Having your foundations in place and your house in order will not only give you newfound visibility right across your organisation; it will also allow better insights and faster decision-making and provide the flexibility to adapt and to scale your data strategy inexpensively as you move proactively towards your future, supported by your data as an organisational asset.
If you’d like to know more about getting your data foundations right, right from the start, or anything to do with good data, done well, we’d love to chat with you.
Make sure you join us for the next instalment in the series for more answers, insights, and tips from Leighton and Alex about how your organisation can utilise your data more effectively, improve your operations and fuel connected customer experiences – good data, done well – for your organisation’s future success.