Data engineer

What does a data engineer do?

A data engineer builds and looks after the systems that move data from where it is created to where people use it.

In Singapore, the role sits in a data or technology team at banks, government agencies, e-commerce firms, logistics companies and start-ups.

Data engineers work with data scientists, analysts and business intelligence teams who need clean, timely data. They also work with software engineers who own the source systems, and with security and compliance staff.

What the work involves

• Write and test pipelines that extract data from source systems, transform it and load it into a warehouse or lake.
• Handle streaming data as well as scheduled batch jobs, depending on how fast the business needs the numbers.
• Meet analysts and product teams to understand what data they need, then choose table designs that make it easy to reach.
• Monitor pipeline runs, find why a job failed or slowed down, and fix it.
• Automate manual data collection, release and reporting steps so they run on a schedule.
• Read about new database and data processing tools and try the promising ones on a small scale.

Pay in Singapore

Back to top

Skills a data engineer needs

Back to top

Data engineer job description

We are looking for a data engineer to design, build and run the data pipelines behind our reporting, analytics and machine learning work.

You will work in a small data team in Singapore, join up data from many internal systems and keep it accurate, secure and on time. You will use SQL and Python every day.

Responsibilities

• Build and maintain batch and streaming pipelines that extract, transform and load data.
• Model data in a warehouse so that analysts can query it without reworking it every time.
• Test pipelines for reliability and for scale as data volumes grow.
• Set up data quality checks and alerts so bad data is caught before it reaches a dashboard.
• Schedule and monitor jobs, and fix failures quickly.
• Connect new data sources and fit them into the existing infrastructure.
• Tune queries and storage to keep run times and cloud costs under control.
• Apply access controls and follow the organisation's data governance and privacy rules.
• Document pipelines and review teammates' code.

Requirements

• A degree in computer science, information systems, engineering or a related field, or a diploma with strong project work.
• Typically 1 to 3 years of experience for a mid-level role; fresh graduates apply for junior roles.
• Strong SQL and working Python.
• Experience with an orchestration tool such as Airflow and a cloud data warehouse such as BigQuery or Snowflake.
• Familiarity with one major cloud platform (AWS, Azure or GCP), Git and Docker.

Back to top

Data engineer career path

Most data engineers start in a junior pipeline role and move to senior work after several years.

Beyond that, you can lead a team or move into architecture.

1. Junior data engineer

   Entry

   Writes and tests parts of pipelines under guidance, and learns the team's tools, SQL standards and code review habits.

2. Data engineer

   Mid · SkillsFuture title

   Builds and maintains pipelines on their own, picks data structures to fit business needs, and automates collection and reporting.

3. Senior data engineer

   Senior · SkillsFuture title

   Designs the data platform, sets standards for quality and governance, and guides other engineers. Also leads projects and deals with stakeholders.

4. Lead data engineer or data engineering manager

   Lead

   Owns the team's plan, hires and coaches engineers, and agrees priorities with business leaders.

Steps marked "SkillsFuture title" follow the career pathway in the SkillsFuture Skills Framework. The others are titles commonly used in Singapore.

Related jobs and their pay

Back to top

Data engineer interview questions

We wrote these questions from the skills this job needs. They are not an official question bank.

1. What is the difference between ETL and ELT, and when would you pick each?

   What a good answer covers

   • ETL transforms data before loading it; ELT loads raw data first and transforms inside the warehouse.
   • ELT suits cloud warehouses with cheap storage and strong compute.
   • ETL still fits when data must be masked or filtered before it lands.
   • Mentions keeping raw data for replay.

2. Explain a star schema. How does it differ from a fully normalised model?

   What a good answer covers

   • Fact tables hold events or measures; dimension tables hold descriptive attributes.
   • Star schemas trade some duplication for simpler, faster queries.
   • Normalised models cut redundancy and suit transactional systems.
   • Gives an example such as orders and customers.

3. A SQL query on a large table has become slow. How do you find the cause and fix it?

   What a good answer covers

   • Reads the query plan to see scans, joins and sorts.
   • Checks partitioning, clustering or indexes.
   • Filters early and selects only needed columns.
   • Considers pre-aggregating or materialising the result.

4. How do you make an Airflow task safe to retry?

   What a good answer covers

   • Makes tasks idempotent so reruns give the same result.
   • Uses the logical run date, not the current time, to pick data.
   • Writes to a staging table, then swaps or merges.
   • Sets retries, timeouts and alerts.

5. Your PySpark job is slow because one task takes much longer than the others. What do you check?

   What a good answer covers

   • Suspects data skew on a join or group-by key.
   • Looks at the Spark UI for uneven partitions.
   • Tries salting keys, broadcast joins or repartitioning.
   • Checks for unnecessary shuffles.

6. How would you design a pipeline that processes events from Kafka and handles late or duplicate messages?

   What a good answer covers

   • Uses event time with watermarks for late data.
   • Deduplicates on a unique event ID.
   • Understands at-least-once delivery and makes sinks idempotent.
   • Explains consumer offsets and replay.

7. What data quality checks would you add to a daily orders table, and where would they run?

   What a good answer covers

   • Checks for nulls, duplicates, valid ranges and row counts against yesterday.
   • Checks that foreign keys match their parent tables.
   • Runs checks after load and before publishing, with a tool such as Great Expectations or dbt tests.
   • Decides which failures block the pipeline and which only alert.

8. When would you choose BigQuery or Snowflake over PostgreSQL for analytics?

   What a good answer covers

   • Warehouses separate storage from compute and scale for large scans.
   • PostgreSQL suits transactional workloads and smaller analytical loads.
   • Considers cost model, concurrency and team skills.
   • Mentions columnar storage.

1. A finance dashboard shows yesterday's revenue as zero. Walk us through how you investigate.

   What a good answer covers

   • Checks whether the pipeline ran and for how long.
   • Traces the data back table by table to find where it dropped.
   • Tells the dashboard owner early and gives an estimate.
   • Fixes the data, then adds a check to catch it next time.

2. A team wants near real-time data, but the current job runs once a night. How do you decide whether to move to streaming?

   What a good answer covers

   • Asks what decision needs the faster data and how fast.
   • Compares the cost and complexity of streaming with more frequent batches.
   • Considers source system limits and data quality.
   • Proposes a small pilot.

3. Two teams define 'active customer' differently and their dashboards disagree. What do you do?

   What a good answer covers

   • Gets both teams to agree one definition and an owner.
   • Puts the logic in one shared model.
   • Documents it in a data catalogue.
   • Explains the change to affected users.

4. You must move a legacy on-premises database to a cloud warehouse with little downtime. How do you plan it?

   What a good answer covers

   • Inventories tables, dependencies and consumers.
   • Runs old and new side by side and compares row counts and totals.
   • Plans a cut-over window and a rollback.
   • Moves low-risk tables first.

1. Tell us about a time you broke production data. What happened and what changed afterwards?

   What a good answer covers

   • Owns the mistake without blaming others.
   • Describes how they found and fixed it.
   • Shows how they told stakeholders.
   • Names the test, review or alert added afterwards.

2. Describe how you explained a technical limit to a non-technical stakeholder who wanted something faster.

   What a good answer covers

   • Uses plain words and a concrete example.
   • Offers options with trade-offs.
   • Agrees on what can be delivered and when.
   • Follows up in writing.

1. Your pipeline loads customer records that include NRIC numbers and phone numbers. How do you handle them under the PDPA?

   What a good answer covers

   • Collects NRIC numbers only when the law requires it or identity must be verified to a high degree, as PDPC guidelines set out.
   • Restricts access and masks or pseudonymises data in non-production.
   • Knows to involve the data protection officer for retention and breach questions.
   • Logs who accessed what.

2. You join a bank in Singapore. What extra care does a data pipeline need in a regulated financial institution?

   What a good answer covers

   • Expects Monetary Authority of Singapore (MAS) guidelines on technology risk and outsourcing to apply.
   • Audit trails, change control and access reviews.
   • Data residency and vendor questions for cloud use.
   • Works with risk and compliance teams before building.

Common questions about a data engineer job

Sources