Data Engineering & Architecture

Data pipelines, data warehousing, ETL, data infrastructure

Data engineering and architecture form the two pillars of data work. Most organisations produce data from many sources: a website, a mobile app, a CRM, a payment processor, an operational database, all of which might be relevant to an insight that might guide decision-making.

Data engineering is the work of building and maintaining the systems that move, transform, and store data so that analysts, data scientists, and business teams can use it. Data engineering is a different discipline to data science. If data science is the work of extracting insight from data, data engineering is the work of making sure clean, reliable and usable data is available in the first place.

Data engineers build the pipelines that extract this data, transform it into a consistent format, and load it into a central warehouse where it can be queried. This process is called ETL (extract, transform, load) or ELT. The data pipeline is a digital assembly line, and acts to provide organisations with clean, consistent data that can be easily analysed.

A data warehouse is a digital place where data is stored in an organised and searchable way. It organises structured data sources like SQL databases, and unify data from multiple sources into one clean and clear location.

Data architecture refers to the way that organisations design and structure their data systems, and the goal is to have efficient collection, storage and processing of data. As well as making sure the organisation has frameworks for these, data architects must also be aware of data security, compliance and governance. It blends strategic IT infrastructure with the organisation’s objectives.