Data Engineering
Data engineering is the work of building and maintaining the systems that move, transform, and store data so that analysts, data scientists, and business teams can use it. If data science is the work of extracting insight from data, data engineering is the work of making sure clean, reliable and usable data is available in the first place. Data engineering is one of the better-paid entry points into the data industry, and demand has historically outstripped supply.
Why SQL is the core skill
Data engineers write a lot of SQL. This means knowledge has to go beyond just SELECT and WHERE. Engineers run complex transformations involving multiple joins, window functions for calculating running totals and rankings across partitions, and CTEs (common table expressions) to break complex logic into readable steps. A data engineer has the ability to write efficient SQL that does not bring a warehouse to its knees.
What to focus on
Get comfortable with window functions: ROW_NUMBER, RANK, LAG, LEAD, and SUM OVER PARTITION BY. Learn what a CTE is and when to use one versus a subquery. Understand query plans: most databases let you run EXPLAIN on a query to see how it will be executed and where it is spending time. This is how you diagnose a slow query.
What Python is used for in data engineering
Python is the scripting language of choice for data pipelines. It gets used to call APIs and extract data, to read and write files in formats such as CSV, JSON and Parquet, to clean and transform data before loading it, and to write the orchestration logic that runs pipelines on a schedule. You do not need deep Python expertise to start, but you need to be comfortable writing scripts that run reliably without manual intervention.
What to practise
A way to check your competence with data engineering tasks is to write a script that calls a public API, extracts some data, transforms it (filter, rename columns, handle missing values), and writes it to a CSV file. Next, write a second script that reads that CSV and loads it into a local database. These two steps represent the core of the ELT process. Once you can do it manually, you can move on to learning the tools that automate and scale it.
What a data warehouse is
BigQuery, Snowflake, and Redshift are the three most common data warehouses in the job market. These are databases optimised for analytical queries rather than transactional ones. A traditional database is designed to handle thousands of small reads and writes per second (for tasks such as looking up a single order or updating a stock level), while a data warehouse is designed to handle queries that scan millions of rows to produce aggregated results.
How they differ from regular databases
Data warehouses use columnar storage, meaning data is stored column by column rather than row by row, so a query that only needs three columns from a table with fifty columns reads a fraction of the data a row-oriented database would. They are also typically separated from the compute used to query them, so storage and compute are independently scalable, which is why they can run large queries cheaply.
What dbt does
dbt (data build tool) is the tool that has most changed how data transformation is done. Before dbt, teams wrote transformation logic in scripts scattered across different places, with no version control, no testing, and no documentation. dbt lets you write your transformations as SQL models in a version-controlled repository, run tests to check data quality, and auto-generate documentation. It has become the standard tool for data transformation in the warehouse.
How to learn it
Start with dbt Core (the free open source version). The official dbt tutorials walk you through the jaffle shop project, which covers the full basics: building models, testing them, and understanding the DAG (directed acyclic graph) that defines the order in which transformations run. After that, find a real dataset to practise on.
What orchestration solves
A data pipeline is a sequence of tasks: extract data from source A, extract data from source B, join and transform them, load the result to the warehouse, run the dbt models. Each task depends on the previous one completing successfully. If extraction fails, the transformation should not run. Orchestration tools manage these dependencies, run pipelines on a schedule, retry on failure, and alert when something goes wrong.
Apache Airflow is the standard
Airflow is the most widely used orchestration tool. You write pipelines as Python code using a directed acyclic graph (DAG) structure. It has a web interface that shows what is running, what has failed, and the history of each run. Understanding Airflow basics is expected in most senior data engineering roles. Alternatives like Prefect and Dagster are worth knowing once you have the fundamentals of Airflow down.
Why data quality is an engineering problem
The worst outcome for analysts is a data pipeline that technically runs successfully but produces wrong numbers, because this is hard to catch. If the pipeline fails, that’s easy to fix, but being able to notice bad data is much harder. Analysts make decisions based on the numbers they have in front of them, and by the time the error is found, it may have had a big impact. This is why data quality is so vital, and the key concern for engineers.
What data quality checks look like
The bare minimum check is a test that primary keys are unique and not null. Test that foreign keys match across tables. Test that metric values are within expected ranges (revenue cannot be negative; user counts should not drop by 90% overnight). dbt tests cover the first two categories. Great Expectations and dbt’s built-in assertions cover the third. Set up alerting so failures are caught before dashboards are wrong for days.
