© 2021 Gartner, Inc. and/or its affiliates. All rights reserved. Gartner is a registered trademark of Gartner, Inc. or its affiliates. This presentation, including all supporting materials, is proprietary to Gartner, Inc. and/or its affiliates and is for the sole internal use of the intended recipients. Because this presentation may contain information that is confidential, proprietary or otherwise legally protected, it may not be further copied, distributed or publicly displayed without the express written permission of Gartner, Inc. or its affiliates.

The AI Engineering JourneyErick Brethenoux, GartnerApril 15, 2021

2 © 2021 Gartner, Inc. and/or its affiliates. All rights reserved.

About Erick

Gartner VP Research & AI Research Agenda lead – Aug 2017 IBM Director of Machine Learning & Decision Management Strategy SPSS VP of Corporate Development Lazard VP, Technology Investments, M&A and VC Gartner, Research Director of Advanced Technologies New Science, Research Director Advanced Technologies and Applications French Embassy, Scientific Mission University of Delaware, PhD Cognitive Science

Years of experience: 3.5 years at Gartner, 30 years industry experience

3 © 2021 Gartner, Inc. and/or its affiliates. All rights reserved.

Key Issue Takeaway:Through 2023, at least 50% of IT leaders will struggle to move their AI predictive projects from proof of concept to production maturity.

4 © 2021 Gartner, Inc. and/or its affiliates. All rights reserved.

Google circa 2014… models in context - still true!

Source: “Machine Learning: The High-Interest Credit Card of Technical Debt” – NIPS 2014 Workshop

5 © 2021 Gartner, Inc. and/or its affiliates. All rights reserved.

In two years organizations have (barely) crossed the 50/50 threshold

Proportion of prototypes that make it to productionPercentage of respondents

53%Average proportion of projects that make it from prototypes to production (fully deployed)

2018 2020

47%

© 2021 Gartner, Inc. and/or its affiliates. All rights reserved. Gartner is a registered trademark of Gartner, Inc. or its affiliates. This presentation, including all supporting materials, is proprietary to Gartner, Inc. and/or its affiliates and is for the sole internal use of the intended recipients. Because this presentation may contain information that is confidential, proprietary or otherwise legally protected, it may not be further copied, distributed or publicly displayed without the express written permission of Gartner, Inc. or its affiliates.

MLOps to ModelOps

7 © 2021 Gartner, Inc. and/or its affiliates. All rights reserved.

KPIs validation

start here

Business Understanding

Data Sources Discovery

Data Wrangling

AnalysisModel Validation

New Data Acquisition

Model Release

End-points identification

Parameters testing

Integration testing

Instantiation validation

re/tuningchallenging

explainingvisualizing

complying

Model activation

Model deployment

Application integration

Model behavior tracking

Production audit procedure

Management & Governance

Establishing a sustainable, repeatable, measurable and continuous operationalization process.

Operationalization cycle / Activation

Development cycle

Operationalization cycle / Release Business drift

Data drift

Mathematical drift

Performance drift

Ethical drift

MLOps Discipline

8 © 2021 Gartner, Inc. and/or its affiliates. All rights reserved.

From MLOps to ModelOps

9 © 2021 Gartner, Inc. and/or its affiliates. All rights reserved.

Demystifying XOps – DataOps, MLOps, ModelOps, Platform Ops for AI

10 © 2021 Gartner, Inc. and/or its affiliates. All rights reserved.

Invariant AI Engineering Themes and Governance Principles

• Security• Ethics• Autonomy• Decisioning• Change Management

11 © 2021 Gartner, Inc. and/or its affiliates. All rights reserved.

Invariant AI Engineering Themes and Governance Principles

• Security• Ethics• Autonomy• Decisioning• Change Management

12 © 2021 Gartner, Inc. and/or its affiliates. All rights reserved.

AI Security

Manipulating AI at Runtime:• Pertubations or ill-intended inputs (data, behavior) • Query attacks to determine model logic• Changes to Model, Operational rules,

External or Insider Attacker

Training Set Dev Set Test Set

Dataset Used for Model Dev Data After Deployment

Assess &Adjust

DevelopD

eplo

y

Production Data

Run

Training Data Poisoning• Change model outcomes

TrainModel

Model Theft at any time • Subsequent Malicious use/ Replacement• Steal intellectual property • Inversion attack

13 © 2021 Gartner, Inc. and/or its affiliates. All rights reserved.

AI Ethics

AI Ethics Guidelines

Fair

Interpretable & Transparent

Human-centric & Socially

Beneficial

Accountable

Secure & Safe

14 © 2021 Gartner, Inc. and/or its affiliates. All rights reserved.

Recommendations

15 © 2021 Gartner, Inc. and/or its affiliates. All rights reserved.

Recommendations

Minimize the technical debt of deployed ML models by monitoring and revalidating their business value by adopting a foundational ModelOps practice – first stage of the AI Engineering discipline.Ensure the integrity, transparency and sustainability of AI models through a systematic process, and enduring governance principles; based on fairness, trustworthiness, reliability and accountability.Identify projects in which a fully data-driven, ML-only approach is unviable and inefficient while exploring the expressive power of Composite AI to solve complex business problems.Extend the skills of AI experts, to cover multiple AI techniques, both from a model development and operationalization perspectives.

16 © 2021 Gartner, Inc. and/or its affiliates. All rights reserved.

ModelOps

Composite AI

Generative AI

Tech

nolo

gy C

ompl

exity

Operational Complexity

The AI Engineering Evolution

Adaptive Systems

AI engineering is a discipline focused on the governance and life cycle management of a wide range of operationalized AI and decision models, providing an uninterrupted flow between the development (human or machine-based), and operationalization of AI models (leveraging one or multiple AI techniques).

(Stage 1)

(Stage 2)

(Stage 3)

© 2021 Gartner, Inc. and/or its affiliates. All rights reserved. Gartner is a registered trademark of Gartner, Inc. or its affiliates. This presentation, including all supporting materials, is proprietary to Gartner, Inc. and/or its affiliates and is for the sole internal use of the intended recipients. Because this presentation may contain information that is confidential, proprietary or otherwise legally protected, it may not be further copied, distributed or publicly displayed without the express written permission of Gartner, Inc. or its affiliates.

Thank you