Site reliability engineer

What does a site reliability engineer do?

A site reliability engineer keeps software systems available, fast and safe to run.

The role applies software engineering to operations work. You write code and tools to replace manual tasks, and you respond when systems fail.

In Singapore, SREs usually sit in platform, infrastructure or engineering teams. Banks, government agencies, e-commerce firms, payment companies and cloud-based startups hire them. The SkillsFuture framework places the role in software engineering, with an SRE manager role above it.

You work with backend developers, product teams, security staff and cloud or network engineers. During outages you also talk to support and business teams, so they know what is affected and when it should be fixed.

What the work involves

• Check dashboards and alerts for latency, errors and capacity, and tune alerts that fire too often.
• Write scripts and tools in Go or Bash to automate deployments, restarts and recovery.
• Respond to incidents: find the cause, restore service and keep stakeholders updated.
• Write post-incident reviews and agree fixes with the developers who own the service.
• Change infrastructure with Terraform and review Kubernetes configuration for new services.
• Review upcoming features with developers so they don't hurt performance or availability.

Pay in Singapore

Back to top

Skills a site reliability engineer needs

Back to top

Site reliability engineer job description

We are looking for a site reliability engineer to keep our production systems available, fast and secure.

You will build monitoring, automation and recovery tools, join the on-call rotation and work with development teams to make services more reliable. You will report to an engineering or platform manager in our Singapore office.

Responsibilities

• Monitor the performance, availability, latency and security of production systems.
• Build dashboards and alerts that point to real problems and cut noise.
• Automate deployments, scaling, restarts and recovery with code and scripts.
• Manage cloud infrastructure as code, using tools such as Terraform.
• Run and improve Kubernetes clusters and the CI/CD pipelines that deploy to them.
• Join the on-call rotation, lead incident response and restore service quickly.
• Write post-incident reviews and track the follow-up actions to completion.
• Set reliability targets with product teams and report on how well services meet them.
• Test disaster recovery plans and check that backups and failover work.
• Review new features and designs for effects on performance and stability.

Requirements

• Diploma or degree in computer science, information technology or a related field, or equivalent work experience.
• Typically 2 to 5 years in software engineering, infrastructure or operations, with some production support.
• Strong Linux skills, shell scripting and at least one language such as Go or Python.
• Hands-on use of Kubernetes, Terraform and a public cloud platform.
• Experience with monitoring tools such as Datadog, Grafana or Prometheus, and with CI/CD pipelines.
• Willingness to join an on-call rotation, including evenings and weekends.

Back to top

Site reliability engineer career path

SREs usually start in software engineering, infrastructure or operations.

The SkillsFuture ladder runs from site reliability engineer to SRE manager, with senior and head-level steps added by many employers.

1. Junior site reliability engineer

   Entry

   You handle alerts, run routine fixes and write small scripts under guidance. You learn the team's systems and join on-call as a secondary responder.

2. Site reliability engineer

   Mid · SkillsFuture title

   You own the monitoring, automation and incident response for several services. You build tools that cut manual work and improve recovery.

3. Senior site reliability engineer

   Senior

   You lead major incidents, set reliability targets with product teams and guide design decisions. You mentor less experienced engineers.

4. Site reliability engineering manager

   Lead · SkillsFuture title

   You manage a team of SREs, plan their work and on-call cover, and report on service health to senior management.

5. Head of platform and reliability

   Head

   You set the reliability and platform direction across engineering and decide on investment in tooling, staffing and risk.

Steps marked "SkillsFuture title" follow the career pathway in the SkillsFuture Skills Framework. The others are titles commonly used in Singapore.

Related jobs and their pay

Back to top

Site reliability engineer interview questions

We wrote these questions from the skills this job needs. They are not an official question bank.

1. Explain SLI, SLO and error budget. Pick a payment API and give an example of each.

   What a good answer covers

   • SLI is a measured value, such as the share of requests that succeed within 300 ms
   • SLO is the target for that value over a period, such as 99.9 percent over 30 days
   • Error budget is the allowed shortfall and guides how fast the team ships changes
   • Chooses measures that reflect what users feel, beyond server health

2. A Kubernetes pod is stuck in CrashLoopBackOff. How do you find the cause?

   What a good answer covers

   • Checks pod status and events with kubectl describe
   • Reads current and previous container logs
   • Looks at resource limits, failed probes, missing config or secrets, and image errors
   • Reproduces the fault outside production if needed
   • Fixes the root cause instead of only restarting the pod

3. How would you organise Terraform code for development, staging and production environments?

   What a good answer covers

   • Uses reusable modules and separate state per environment
   • Stores state remotely with locking
   • Reviews terraform plan output in pull requests before apply
   • Keeps secrets out of code and state where possible
   • Describes how to detect and handle drift from manual changes

4. A Linux server shows high load average but low CPU use. What do you check?

   What a good answer covers

   • Considers processes waiting on disk or network as well as CPU
   • Uses tools such as top, vmstat, iostat and ss
   • Checks memory pressure, swap and disk latency
   • Looks for blocked processes and noisy neighbours on shared hosts
   • Confirms the cause before changing anything

5. Design alerting for a customer-facing web service. How do you avoid alert fatigue?

   What a good answer covers

   • Alerts on symptoms users feel, such as error rate and latency, before internal causes
   • Sets thresholds from SLOs and uses burn-rate alerts
   • Separates pages that wake someone from tickets that can wait
   • Links each alert to a runbook
   • Reviews noisy alerts regularly and removes or tunes them

6. Write a shell script that reads a web access log and lists the ten endpoints with the most 5xx errors.

   What a good answer covers

   • Uses awk, sort and uniq or equivalent in a clear pipeline
   • Handles malformed lines and an empty file
   • Quotes variables and sets safe options such as set -euo pipefail
   • Takes the log path as an argument
   • Explains how they would test it on sample data

7. Design a CI/CD pipeline that deploys to Kubernetes with a safe rollback.

   What a good answer covers

   • Builds, tests and scans the image before deploy
   • Uses a canary or rolling release with health checks
   • Rolls back automatically when error rate or latency crosses a threshold
   • Separates build artefacts from environment configuration
   • Keeps an audit trail of who deployed what and when

8. You must write a Go tool that checks the health of 500 endpoints every minute. How do you build it?

   What a good answer covers

   • Uses goroutines with a bounded worker pool
   • Sets timeouts with context so slow endpoints don't block the run
   • Collects results through channels and handles errors
   • Avoids data races and tests with the race detector
   • Exposes results as metrics for the monitoring system

1. At 9 am on a Monday, latency jumps on your main service after a weekend deployment. Walk us through your first 15 minutes.

   What a good answer covers

   • Confirms the impact from dashboards and declares an incident
   • Checks recent deployments and rolls back if the timing matches
   • Assigns roles, such as incident lead and communications
   • Posts a status update to stakeholders
   • Collects evidence for the later review while restoring service

2. A product team wants to release on Friday afternoon, but the service has used up its error budget this month. What do you do?

   What a good answer covers

   • Shows the data on budget use and recent incidents
   • Proposes options such as a smaller release, a feature flag or a delay
   • Explains the risk in business terms the team can weigh
   • Agrees the rule in advance so the team doesn't argue it case by case
   • Keeps the relationship with the product team positive

3. You use Claude Code to draft a Terraform change for a production network. How do you check it before it is applied?

   What a good answer covers

   • Reads every line of the generated code and checks it before trusting it
   • Runs plan and reviews what will be created, changed or destroyed
   • Gets a peer review through the normal pull request process
   • Tests in a non-production environment first
   • Keeps secrets and company data out of prompts

1. Describe an incident you handled. What did you do and what changed afterwards?

   What a good answer covers

   • Gives a clear sequence of events and their own role
   • Explains how they communicated during the incident
   • Names the root cause without blaming a person
   • Describes follow-up actions that were completed and their effect

2. Tell us about a time you reduced repetitive manual work for your team.

   What a good answer covers

   • Describes how they spotted the task and measured the time it took
   • Explains the tool or script they built and why
   • Shows how others adopted it and who maintains it
   • States the result, such as fewer errors or hours saved

1. Application logs may contain customer names, NRIC numbers or phone numbers. How do you handle this under the Personal Data Protection Act (PDPA)?

   What a good answer covers

   • Avoids logging personal data unless needed and masks or removes it where possible
   • Limits who can read logs and sets retention periods
   • Protects logs in transit and at rest
   • Raises a suspected data breach with the data protection officer quickly
   • Says they would check company policy and PDPA guidance for exact duties

2. You support a bank regulated by the Monetary Authority of Singapore (MAS). A major outage occurs. How do you prepare for the regulator's expectations?

   What a good answer covers

   • Knows regulated firms must report serious technology incidents within a time the regulator sets
   • Keeps an accurate timeline and impact record during the incident
   • Involves risk and compliance teams early
   • Completes a root cause analysis with tracked fixes
   • Regularly tests recovery and failover plans

Common questions about a site reliability engineer job

Sources