Partnership Announcement
6th October 2023
Scope Systems and PeopleTray Announce Strategic Partnership to accelerate the delivery of workforce management software to the mining and mining contracting market.
Read More
Home / Blog / Modern Enterprise Data Architecture – Data Lake or Data Warehouse
Modern Enterprise Data Architecture – Data Lake or Data Warehouse?
Author: Joseph Bojilov, Data Architect
Published: 5th April 2022
In today’s world data is the catalyst for all decision making. In previous years companies needed Data Analysts to read and understand the data collected. Now technology has evolved beyond that, with data being transformed using Business Intelligence solutions, allowing data to be presented in a format that can be understood by most employees.
When we talk about Big Data we usually refer to a Hadoop Ecosystem (technology systems). Hadoop uses programming models to process large data sets across clusters of computers, being able to be scaled from a single server to hundreds of machines. One limitation of Hardoop is that doesn’t fit nicely with Cloud services, it is more suited to physical hardware rather than the cloud.
For data to be transformed into understandable information it first needs to be stored somewhere accessible. The two data solutions that are commonly used for data management are Data Warehouses and Data Lakes, each solution having its own benefits.
Example of Contemporary Enterprise Data Architecture
Data Lakes
Big Data is stored on the Corporate Data Lakes, a logical construct in which data can be stored and further manipulated. They are a vast pool of raw data that isn’t purposely collected for a defined purpose.
Data Lakes can be based on different technologies from File Storage to RDBMS. They are flexible and can store all kinds of data including structured, unstructured, and semi-structured. Structured and semi-structured data includes JavaScript Object Notation (JSON) text, CSV files and website logs. An enterprise data lake supports storage of Big Data for real-time analysis and has more flexibility than the other common data solution, Data Warehouse.
Data Warehouse
An Enterprise data warehouse (EDW), is a centralized repository of data where organisations store data from business systems and other sources in a structured format. It is a subset of organisational Data used for reporting needs, storing structured data only and is not as flexible as Corporate Data Lakes. EDW Data sources can include Online Transaction Processing (OLTP) databases, Customer Relationship Management (CRM) and Enterprise Resource Planning (ERP) databases. Due to data warehouses structured format they cannot be used for Big Data.
Although Enterprise Data Warehouses are less flexible then Data Lakes and are sometimes seen as two separate data repositories, EDW data is subset of Data Lakes. A well-designed and defined Data Lakes able to support the EDW functionality.
Future of Big Data
The increased technology used by companies today such as AI and the emergence of the metaverse means that data solutions will be reimagined for the future in order to store and process new types of data.
One new concept is Data Mesh, a decentralised approach towards data ownership, allowing users to access the data, without it needing to go into a Data Lake or Data Warehouse. However for it to be usable as a data solution it requires a lot of standardisation throughout Enterprise and beyond. While it is a good approach from practical point of view, it will take some time before the Data Mesh concept could be implemented, particularly on an enterprise scale. In the near future I believe that Data Lake and Data Warehouse will be gradually merge into one concept in the future – a Corporate Lake–Warehouse. The Corporate Lake-Warehouse will provide the flexibility of storing unstructured, semi structured and structured data of a data lake, combined with the data warehouses structured collection of reporting data.
While the future of Data Architecture will rely on the requirements of new technology, one thing that is guaranteed is that we are heading towards an even more data driven future – and companies need to be prepared.
Discover more.
Scope Systems Parked Invoice System
26th July 2023
The Parked Supplier Invoice Approval System is a Pronto Xi enhancement created by Scope Systems that provides you with additional functionality by adding the ability to edit and include an approval process for invoices.
Read More
The Mining Outlook According to RIU
14th March 2023
The RIU Explorers conference this year saw a packed full house of over 1800 delegates and 200 exhibitors in the Esplanade Hotel, Fremantle. Both exhibitors and attendees there had a pretty bullish outlook, despite the repeated doom and gloom in the media.
Read More
Workforce Engagement Through Digital Automation
29th November 2022
With staff retention a key issue currently facing the mining industry, discover why digital automation is key to increasing you workforce engagement.
Read More
Top Risks in the Mining Industry 22/23
13th October 2022
The mining industry has once again faced many challenges this year, with a major one again being the microscope the industry has been put under for its male-dominated culture. This year’s Australian Mining Risk Forecast has been recently released by KPMG and it’s an interesting read on what is top of mind for mining executives.
Read More
Going for Gold – The Year in Review
18th August 2022
Take a look at the year in review. From the RIU Explorers conference to Diggers and Dealers, our Sales and Marketing Manager Sonia Turner give her summary of the financial year.
Read More
What is e-invoicing?
8th August 2022
E-invoicing isn’t a new idea, the first electronic invoices were sent over 30 years ago through an electronic data interchange. Since then, e-invoicing has developed into a safe and secure way for suppliers to automatically and digitally exchange invoice information to customers through a secure network. So why move to e-invoicing and what are the benefits it can give to your company?
Read More
ERP Software Solutions for Exploration Mining
28th July 2022
Starting up an exploration venture requires careful planning of capital, with keeping costs to a minimum one of the highest priorities for a company. One aspect of keeping costs low can include the business systems designed to help control expenditure.
Read More
Why Partner with a Software Solutions Provider?
12th July 2022
Why partner with a software solutions specialist? Is it worth considering or are you better off managing on our own? To answer these questions, let’s look at what a specialist like Scope Systems can bring to the table.
Read More
Why Integrate your ERP Software?
28th June 2022
Companies today are more diverse then ever, finding they to have multiple software solutions to add extra functionality to their system. Having an integrated solution allows all your systems to 'talk' to each other , with your ERP acting as your primary system.
Read More
3 Benefits of Software Training
14th June 2022
For some companies, the idea of training staff to use new or existing software is a no brainer, recognising the benefits to the company from the outset. Others are more hesitant, with questions such as if there will be any tangible benefit, what will the true ROI be and what’s stopping staff jumping ship to another company after.
Read More
.Au Domain Names Have Arrived
30th May 2022
From March 2022 a new domain name category has become available for Australian domain names, with the new, shorter .au name arriving.
Read More
Improve Your Budgeting and Forecasting
24th May 2022
Budgeting is one of those unavoidable tasks that most of us would prefer not to do, investing in a budgeting solution can help improve your processes.
Read More
Improving Your Passwords Strength
4th May 2022
A password is the first line of defence in protecting not only your own personal data, but also confidential company data. Check out our tips for creating a strong and memorable password.
Read More
What You Need to Know About STP Phase 2
19th April 2022
Are you ready for the upcoming reporting changes for STP Phase 2? Learn what you need to do now to prepare and what to expect.
Read More
Modern Enterprise Data Architecture – Data Lake or Data Warehouse?
5th April 2022
For data to be transformed into understandable information it first needs to be stored somewhere accessible, most commonly in a Data Warehouse or Data Lake.
Read More
Greater Visibility Across Mining Supply Chains Will Reduce Supplier Risk
28th March 2022
With accurate inventory data, you can measure supplier performance and help reduce extended lead times across your mining supply chain.
Read More
Is Your Tenement Management System Time Consuming?
24th March 2022
Ensuring your multiple tenements activities are being recorded and reported on is vital, as failing to do so can affect your whole mining operation and lead to costly penalties.
Read More
Intelligent Inventory Management Solutions will Transform the Mining Industry
14th March 2022
An inventory management solution will give mining companies data-driven insights to help increase profits by focusing on having the right stock to meet demand.
Read More
How Your ERP System can Help Relieve Worker Shortages
8th March 2022
With shortages of skilled workers across many industries, optimising your ERP system can reduce the stress caused by gaps in your workforce.
Read More
