As artificial intelligence rapidly reshapes industries from healthcare to finance, conversations about ethics and governance have often remained high-level, leaving a gap between principle and practice. Sumit Sinha, a seasoned engineering leader with over 16 years of experience architecting large-scale platforms and advancing conversational AI, argues that trust in AI begins not with policy but with design. From data validation and preprocessing to prompt handling and real-time monitoring, he emphasizes that engineering decisions made at the system level determine whether AI delivers on its promise responsibly. In this interview, Sinha shares how software design can turn abstract ethical principles into actionable safeguards, explores the evolving responsibilities of engineers in high-stakes domains, and offers a roadmap for integrating responsibility into innovation without slowing progress.
Many conversations about AI ethics stay abstract. From an engineering perspective, what are the most immediate design choices, such as data validation or prompt handling, that directly influence whether an AI system can be trusted?
Building trust in AI starts with design decisions. From the engineering side, this means embedding transparency, accountability, and reliability directly into the system architecture. Rather than treating trust as an afterthought, it should guide how data is selected, how outputs are generated, and how human oversight is built in.
The following are some key engineering choices that contribute to making an AI system trustworthy:
- The most foundational choice is what data the system is trained on. Engineers must move beyond simply gathering “more data” and focus on data validation and curation to ensure quality.
- Logging the specific data points that influenced a decision to create an audit trail. This allows for human review and correction, making the system from a “black box” to a transparent tool.
- Providing and tracking a confidence score, and then making decisions only if a specific threshold is met. If a task cannot be performed confidently, make sure there is a way to escalate to a human.
- Designing and testing to make sure that the AI platform only sticks to the provided material and data, and refuses to answer anything that is not covered in the provided data, API or content.
Responsible AI is often framed as a governance problem, but you argue that it actually starts with software design. Can you provide an example of how better engineering practices prevented or reduced bias in a deployed system?
While governance provides the policies and rules, software design provides the concrete design and implementation plan. Policies are only effective if they can be translated into actionable engineering practices. Ethical principles like fairness, transparency, and accountability are abstract until they are coded into a system using software design
One of the examples of this is the choice of data that is used to train the AI model. As an engineering practice, removing specific data points to remove any bias, like name, age-related data, educational background, work history etc., which is relevant to the decision making, helps reduce bias. I have seen this working in various AI models, which have helped with fair decision-making.
Data quality is often referred to as the foundation of AI. How should engineers approach validation and preprocessing to ensure that models don’t inherit systemic errors or skewed results from the start?
Before preprocessing, engineers must validate the raw data to understand its integrity and inherent characteristics. A few validation steps that should help:
- Data completeness – Filtering for missing values (like null or blank space) and also correcting inconsistent values (like for state Florida and FL) helps a lot.
- Outlier and Anomaly detection – Identifying data points that deviate too much from the norm or baseline should be documented, and a decision will need to be made to keep or remove. These anomalies can be due to an error or a rare valid event.
- Bias Detection – Figuring out if a data set is not skewed because the data collection represents only a few groups who are similar is critical to make sure AI is not being trained with only a narrow perspective. For example, checking if the representation of different genders, ethnicities, or socioeconomic groups is proportional to the target population will give a clear picture.
After validation, some pre-processing steps can help:
- Re-Sampling – Removing some over-represented groups and duplicating some under-represented group may help
- Handling missing data – Rather than removing missing data that can potentially cause data loss if a large amount of data is missing, a decision can be made to either use average data or use a more sophisticated method to predict what that missing value could be.
Prompt design is emerging as a new discipline. What role does it play in responsible AI, and how can teams avoid prompts that accidentally encourage harmful or misleading outputs?
Prompt design plays a critical role in responsible AI because it serves as the most immediate and direct control mechanism for an AI’s behavior. While a model’s underlying architecture and training data set its foundational capabilities and biases, the prompt is the final instruction that shapes a specific output. Poorly designed prompts can accidentally encourage harmful or misleading outputs by being ambiguous, manipulative, or exploiting the model’s vulnerabilities. A few things that teams can do to avoid that situation:
- Behavioral Guardrails – Prompts can be used to instruct a model on what it should and shouldn’t do. For example, a system prompt for a chatbot can explicitly forbid it from generating hateful content, regardless of a user’s request.
- Proactive Refusal – Providing instruction to the AI to refuse to answer certain types of questions, like unethical, dangerous etc., will make sure the AI is proactively staying away from these topics, and if asked, it will refuse to answer.
- Making sure that AI understands its system directive and is not confusing that with user input is one of the most important checks teams should perform.
Governance frameworks are essential, but many companies struggle to make them practical. What engineering-level checks or controls can help translate governance principles into day-to-day development workflows?
Some of the engineering-level checks that can be very helpful will be:
- Dashboards – Setting up dashboards to track model performance and also day-to-day variance will be very helpful and can help warrant research and troubleshooting when needed.
- Error detection – Alerting when error rates are higher in general or if higher for a specific user group, will help engineering teams to act quickly to solve any underlying issues
- Stress Testing – Testing for performance and also for the quality by automating negative test scenarios will help in being confident with any new change to the AI model.
System monitoring is sometimes treated as an afterthought. Why is real-time oversight through logging, audits, or drift detection so crucial to building long-term trust in AI systems?
The reason is very simple: an AI model’s performance and ethical behavior are not static. A system that works perfectly in a test environment can fail in the real world, and what is fair today may become biased tomorrow. It is crucial to address the dynamic nature of AI. A few things to keep in mind on this:
- Proactive Detection – Real-time monitoring and alerting will help catch issues sooner as they happen
- Accountability and Explainability – Logging and auditing help create and maintain a record of system decisions with an explanation of why that decision was made. This helps in making sure the system is fair to everyone and is able to provide evidence of that.
- Reliability – Setting up acceptable, defined performance levels and then real-time monitoring will help in making sure that the AI model is performing at the defined level by the engineers at all times.
Ethical AI is often perceived as hindering innovation. How can integrating responsible design practices early in development actually reduce risk, cut costs, and speed up deployment over time?
In my opinion, a proactive approach to responsible AI is a strategic advantage that can reduce risk, cut costs, and accelerate development and deployment. Here are a few things that it helps with:
- Reduced risk and cost of Failure
- Avoiding reputational damage – The cost of public AI failure can be high, which can go beyond the initial investment in ethical AI.
- Legal and regulatory compliance – Being in compliance from the get-go for newly set rules by different governments can help save a lot of time and money in rework.
- Speed of Deployment
- Streamline auditing – Documenting technical workflows from the start will help in making auditing efficient, which helps in making reviews quicker.
- Building with reusable, responsible AI components will make it easier to replicate the standards.
Looking ahead, what new responsibilities will engineers carry as AI becomes more deeply embedded in industries like healthcare, finance, and manufacturing and how should they prepare to meet those challenges?
As AI becomes deeply embedded in high-stakes industries, engineers will carry a new set of responsibilities that go far beyond traditional coding. They will be at the forefront of managing not just technological complexity, but also societal and ethical risk. Some new responsibilities that they will need to carry:
- Risk and Safety Engineering: An AI system’s error in today’s world may have life-or-death consequences, especially in healthcare. Engineers will be responsible for defining and testing for failure modes in a way that is not currently standard practice. This includes designing AI with built-in safeguards like confidence scores, human-in-the-loop overrides, and warning signals when the system encounters data or a scenario it hasn’t seen before.
- Causal Reasoning and Explainability: Engineers will be responsible for building systems that can not only make a prediction but also explain the causal reasoning behind it. For example, a financial model that flags a transaction as fraudulent must be able to explain why it’s suspicious and document it, rather than just giving a risk score. This requires a shift from focusing on predictive accuracy alone to prioritizing interpretation, reasoning, and audit.
To prepare for these challenges, engineers may start focusing on integrated coursework on ethics, public policy, and domain-specific knowledge, which will be helpful as a supplement to the more traditional computer science coursework. Engineers should also seek out certifications and training programs that cover topics like AI safety, algorithmic fairness, and data privacy regulations. Additionally, gaining exposure to work in multidisciplinary teams (such as data science, legal, and fraud) is highly beneficial.
