Inference engineer

What does an inference engineer do?

An inference engineer makes trained AI models run fast, reliably and at a sensible cost in production.

The work starts after data scientists or ML engineers have a model that works. You turn it into a service that real users and systems can call.

In a Singapore organisation, you usually sit in an AI, data or platform team. That could be at a bank, a government agency, a logistics firm, a start-up or a technology company. Some teams run their own GPUs. Others use cloud regions in Singapore.

You work with machine learning engineers, data scientists, backend engineers, site reliability engineers and product managers. You also talk to security and compliance staff when a model handles personal or financial data.

What the work involves

• Measure how long a model takes to answer and how many requests it handles at once, then find the slow parts.
• Shrink or speed up models with quantisation, batching and compiled code, and check the answers stay good enough.
• Package models into services and deploy them on Kubernetes with the right GPU or CPU resources.
• Build dashboards and alerts in tools such as Datadog or Grafana to watch latency, errors and GPU use.
• Test new model versions on a small share of traffic and roll them back if results get worse.
• Work with data scientists and product teams to agree speed, quality and cost targets.

Pay in Singapore

Back to top

Skills an inference engineer needs

Back to top

Inference engineer job description

We are hiring an inference engineer to run machine learning and large language models in production for users in Singapore.

You will take models from the research team, make them fast and affordable, and keep them stable. You will write Python and some C++, deploy on Kubernetes, and monitor live services. You will work with ML engineers, backend engineers and product managers.

Responsibilities

• Deploy trained models as low-latency services behind internal or customer-facing APIs.
• Profile model code and hardware use to find bottlenecks in memory, compute and data transfer.
• Apply quantisation, batching, caching and compilation to cut response time and GPU cost.
• Run and tune open-source models from Hugging Face, including loading, tokenising and versioning.
• Write and maintain Kubernetes configuration, including GPU scheduling and autoscaling rules.
• Set up monitoring, logging and alerts for latency, errors, throughput and resource use.
• Run load tests and A/B or canary releases before new models reach all users.
• Fix production bugs and write short incident reports with follow-up actions.
• Work with data scientists to keep model quality steady when changing how models are served.
• Follow the organisation's data protection and security rules, including the Personal Data Protection Act (PDPA).

Requirements

• Degree in computer science, engineering, mathematics or a related field, or equivalent work experience.
• Typically 2 to 5 years of experience in software or machine learning engineering for mid-level roles.
• Strong Python and PyTorch skills, plus working knowledge of C or C++ for performance-sensitive code.
• Hands-on experience with Kubernetes, containers and cloud services, ideally with GPU workloads.
• Experience with monitoring tools such as Datadog or Grafana and with MLOps practices like model versioning and staged releases.
• Familiarity with Hugging Face models and libraries is often asked for.

Back to top

Inference engineer career path

Most inference engineers start in software or machine learning engineering and move towards platform and performance work.

Titles vary by employer, so the steps below are common ones in Singapore.

1. Junior machine learning engineer

   Entry

   You deploy models using existing pipelines, write tests and help monitor services under guidance.

2. Inference engineer

   Mid

   You own the serving of one or more models, profile performance and run safe releases on your own.

3. Senior inference engineer

   Senior

   You design serving systems, set latency and cost targets, and guide others on optimisation.

4. Lead ML platform engineer

   Lead

   You lead a small team, choose serving tools and hardware, and plan capacity with finance and product teams.

5. Head of ML platform

   Head

   You set the technical direction for how the organisation runs AI in production and manage budgets and vendors.

The titles are ones commonly used in Singapore; employers may name the levels differently.

Related jobs and their pay

Back to top

Inference engineer interview questions

We wrote these questions from the skills this job needs. They are not an official question bank.

1. What is the difference between latency and throughput in model serving, and how does batching affect each?

   What a good answer covers

   • Latency is time per request; throughput is requests or tokens per second.
   • Larger batches raise throughput and GPU use.
   • Batching can raise latency because requests wait to be grouped.
   • You set a maximum wait time based on the product's latency target.
   • You measure both with load tests.

2. How does continuous batching work when serving a large language model, and why does it help?

   What a good answer covers

   • New requests join a running batch as others finish, instead of waiting for a full batch.
   • It keeps the GPU busy when output lengths differ.
   • It improves throughput and cuts waiting time.
   • It needs careful management of the key-value cache memory.
   • You can name a serving tool that supports it.

3. A 7-billion-parameter model does not fit on your single GPU. What options do you consider?

   What a good answer covers

   • Quantise weights to 8-bit or 4-bit and check quality loss.
   • Reduce context length or batch size to limit cache memory.
   • Use a smaller or distilled model if quality allows.
   • Split the model across several GPUs if needed.
   • Compare options on a fixed test set before choosing.

4. How would you find out why a PyTorch model is slow in production?

   What a good answer covers

   • Use a profiler to see time per operation on CPU and GPU.
   • Check for a model left in training mode or gradients left on.
   • Look for CPU-to-GPU copies and forced synchronisation.
   • Check kernel launch overhead and try compilation or fused operations.
   • Re-run the profile after each change to confirm the gain.

5. When would you write a C++ extension or custom kernel instead of staying in Python?

   What a good answer covers

   • When profiling shows a specific hot spot that existing libraries do not fix.
   • When Python overhead dominates for small, frequent operations.
   • You weigh the gain against maintenance and build complexity.
   • You test the new code against the Python version for matching outputs.
   • You keep a fallback path.

6. How do you set up autoscaling for a GPU inference service on Kubernetes?

   What a good answer covers

   • Request GPU resources explicitly and use dedicated node pools.
   • Scale on queue depth or request rate as well as CPU use.
   • Account for slow start while model weights load.
   • Use readiness probes so traffic only reaches loaded pods.
   • Keep a minimum number of replicas to avoid cold starts at peak times.

7. Which metrics and alerts would you set up for a live model endpoint?

   What a good answer covers

   • Latency at median and tail percentiles, such as p95 and p99.
   • Error rate and timeouts.
   • GPU use, memory and queue length.
   • Tokens or requests per second.
   • Dashboards in Grafana or Datadog, with alerts tied to user impact.

8. How do you handle many concurrent requests in a Python inference API?

   What a good answer covers

   • Explain the global interpreter lock and where it matters.
   • Use async handling for network work and worker processes for compute.
   • Move heavy work to the GPU or a dedicated model server.
   • Limit queue size and return clear errors when overloaded.
   • Load test to find the real limit.

1. After a release, p99 latency doubles but average latency looks normal. How do you investigate?

   What a good answer covers

   • Compare traces and metrics before and after the release.
   • Check for long inputs, batch changes or cache misses in the slow requests.
   • Look at GPU memory pressure and pod restarts.
   • Roll back if users are affected, then debug in a test setup.
   • Add an alert on tail latency so you see it sooner next time.

2. Your manager wants a large cut in GPU spending without users noticing a drop in quality. What do you do?

   What a good answer covers

   • Record current cost, latency and quality as a baseline.
   • Try quantisation, better batching, caching and right-sized hardware.
   • Consider a smaller or distilled model for easy requests.
   • Check quality on a fixed test set and with a small live trial.
   • Report savings and risks without jargon.

3. A data scientist hands you a new model version. How do you release it safely?

   What a good answer covers

   • Run offline checks on speed and output quality first.
   • Send a copy of live traffic to it without showing results to users.
   • Release to a small share of users and watch key metrics.
   • Set clear rollback rules in advance.
   • Keep the old version running until the new one is stable.

1. Tell me about a time you gave up some accuracy for speed. How did you get agreement?

   What a good answer covers

   • Describe the speed target and why it mattered.
   • Show the numbers on accuracy loss and speed gain.
   • Explain how you involved the model owner early.
   • Say what you would do differently now.

2. Describe a production incident you helped fix. What changed afterwards?

   What a good answer covers

   • State the impact on users and how you found out.
   • Explain your steps in order, including who you told.
   • Describe the root cause without blaming a person.
   • Name the alert, test or process you added.

1. Your service reads customer messages for a Singapore bank. How do you handle logging and data under the PDPA and MAS expectations?

   What a good answer covers

   • Avoid logging raw personal data; mask or drop it.
   • Set short retention periods and limit who can see logs.
   • Use the data only for the stated purpose.
   • Check where data and models are hosted and any transfer limits.
   • Follow the bank's own Monetary Authority of Singapore (MAS) technology risk and outsourcing rules, and ask compliance when unsure.

2. A Singapore government agency needs its model served on approved infrastructure only. How does that change your design?

   What a good answer covers

   • Confirm which cloud or on-premise environments are approved.
   • Plan for limited internet access, so weights and packages come from internal sources.
   • Check GPU availability and lead times.
   • Build offline monitoring and logging inside the same environment.
   • Document the setup for security review.

Common questions about an inference engineer job

Sources