Causal Artificial Intelligence – The Next Step in Effective Business AI

JUDITH S. HURWITZ is the chief evangelist at Geminos Software, a causal AI platform company. For more than 35 years she has been a strategist, technology consultant to software providers, and a thought leader having authored 10 books in topics ranging from augmented intelligence, data analytics, and cloud computing.
JOHN K. THOMPSON is an international technology executive with over 37 years of experience in the fields of data, advanced analytics, and artificial intelligence (AI). John is responsible for the global AI function at EY. He has previously led the global Artificial Intelligence and Rapid Data Lab teams at CSL Behring and is the bestselling author of three books on data analytics.

Foreword xix Preface xxiii Introduction xxix Chapter 1 Setting the Stage for Causal AI 1 Why Causality Is a Game Changer 2 Causal AI in Perspective with Analytics 7 Analytical Sophistication Model 8 Analytics Enablers 10 Analytics 10 Advanced Analytics 11 Scope of Services to Support Causal AI 11 The Value of the Hybrid Team 13 The Promise of AI 14 Understanding the Core Concepts of Causal AI 15 Explainability and Bias Detection 15 Explainability 17 Detecting Bias in a Model 17 Directed Acyclic Graphs 18 Structural Causal Model 19 Observed and Unobserved Variables 20 Counterfactuals 21 Confounders 21 Colliders 22 Front- Door and Backdoor Paths 23 Correlation 24 Causal Libraries and Tools 25 Propensity Score 25 Augmented Intelligence and Causal AI 26 Summary 27 Note 27 Chapter 2 Understanding the Value of Causal AI 29 Defining Causal AI 30 The Origins of Causal AI 33 Why Causality? 34 Expressing Relationships 37 The Ladder of Causation 38 Rung 1: Association, or Passive Observation 40 Rung 2: Intervention, or Taking Action 40 Rung 3: Counterfactuals, or Imagining What If 42 Why Causal AI Is the Next Generation of AI 43 Deep Learning and Neural Networks 43 Neural Networks 44 Establishing Ground Truth 45 The Business Imperative of a Causal Model 46 The Importance of Augmented Intelligence 51 The Importance of Data, Visualization, and Frameworks 52 Getting the Appropriate Data 52 Applying Data and Model Visualization 55 Applying Frameworks After Creating a Model 56 Getting Started with Causal AI 57 Summary 58 Notes 59 Chapter 3 Elements of Causal AI 61 Conceptual Models 62 Correlation vs. Causal Models 63 Correlation- Based AI 63 Causal AI 63 Understanding the Relationship Between Correlation and Causality 64 Process Models 66 Correlation- Based AI Process Model 67 Causal- Based AI Process Model 69 Collaboration Between Business and Analytics Professionals 72 The Fundamental Building Blocks of Causal AI Models 75 The Relations Between DAGs and SCMs 76 Explaining DAGs 76 Causal Notation: The Language of DAGs 78 Operationalizing a DAG with an SCM 79 The Elements of Visual Modeling 81 Nodes 83 Variables 83 Endogenous and Exogenous Variables 83 Observed and Unobserved Variables 84 Paths/Relationships 84 Weights 86 Summary 88 Notes 89 Chapter 4 Creating Practical Causal AI Models and Systems 91 Understanding Complex Models 92 Causal Modeling Process: Part 1 94 Step 1: What Are the Intended Outcomes? 95 Step 2: What Are the Proposed Interventions? 97 Step 3: What Are the Confounding Factors? 99 Step 4: What Are the Factors Creating the Effects and Changes? 102 Common/Universal Effects in a Causal Model 102 Refined Effects in a Causal Model 103 Step 5: Creating a Directed Acyclic Graph 105 Step 6: Paths and Relationships 105 Types of Paths 106 Path Connecting an Unobserved Variable 107 Front- Door Paths 108 Backdoor Paths 108 Modeling for Simplicity to Understand Complexity 109 Step 7: Data Acquisition 110 Causal- Based Approach: Part 2 112 Step 8: Data Integration 113 Step 9: Model Modification 114 Step 10: Data Transformation 115 Step 11: Preparing for Deployment in Business 118 Summary 121 Notes 122 Chapter 5 Creating a Model with a Hybrid Team 125 The Hybrid Team 126 Why a Hybrid Team? 127 The Benefits of a Hybrid Team 128 Establishing the Hybrid Team as a Center of Excellence 129 How Teams Collaborate 131 But Why? 132 Defining Roles 134 Leaders and Business Strategists 137 Subject- Matter Experts 138 Data Experts 140 Software Developers 142 Business Process Analysts 143 Information Technology Expertise 143 Project Manager(s) 144 The Basics Steps for a Hybrid Team Project 145 An Overview of Model Creation 146 It Depends on Your Destination 150 Understanding the Root Cause of a Problem 151 Understanding What Happened and Why 153 The Importance of the Iterative Process 154 Summary 155 Chapter 6 Explainability, Bias Detection, and AI Responsibility in Causal AI 157 Explainability 158 The Ramifications of the Lack of Explainability 159 What Is Explainable AI in Causal AI Models? 161 Black Boxes 162 Internal Workings of Black-Box Models 162 Deep Learning at the Heart of Black Boxes 163 Is Code Understandable? 163 The Value of White-Box Models 166 Understanding Causal AI Code 167 Techniques for Achieving Explainability 169 Challenges of Complex Causal Models 169 Methods for Understanding and Explaining Complex Causal AI Models 171 The Importance of the SHAP Explainability Method 172 Detecting Bias and Ensuring Responsible AI 175 Bias in Causal AI Systems 176 Responsible AI: Trust and Fairness 178 How Causal AI Addresses Bias Detection 180 Tools for Assessing Fairness and Bias 182 The Human Factor in Bias Detection and Responsible AI 183 Summary 184 Note 184 Chapter 7 Tools, Practices, and Techniques to Enable Causal AI 185 The Causal AI Pipeline 187 Define Business Objectives 190 Model Development 193 Data Identification and Collection 195 Data Privacy, Governance, and Security 197 Synthetic Data 198 Model Validation 199 Deployment/Production 201 Monitor and Evaluate 203 Update and Iterate 205 Continuous Learning 208 The Importance of Synthetic Data 210 Why Create Synthetic Data? 210 Overcoming Data Limitations 211 Enhancing Data Privacy and Security 211 Model Validation and Testing 211 Expanding the Range of Possible Scenarios 212 Reducing the Cost of Data Collection 212 Improving Data Imbalance 213 Encouraging Collaboration and Openness 213 Streamlining Data Preprocessing 213 Supporting Counterfactual Analysis 213 Fostering Innovation and Experimentation 214 Creating Synthetic Data 214 Generative Models 214 Agent-Based Modeling 215 Data Augmentation 215 Data Synthesis Tools and Platforms 215 Conditional Synthetic Data Generation 216 Synthetic Data from Text 216 The Limitations of Synthetic Data 217 Current State of Tools and Software in Causal AI 218 The Role of Open Source in Causal AI 218 Commercial Causal AI Software 221 CausaLens 221 Geminos Software 223 Summary 223 Chapter 8 Causal AI in Action 225 Enterprise Marketing in a Business- to- Consumer Scenario 226 DDCo Marketing Causal Model: Annual Pricing Review and Update Cycle 228 Incorporating Internal and External Factors in the Model and DAG 230 Easily Enabling Iterating 231 End-User-Driven Exploration 232 Bench Testing 234 DDCo Marketing Causal Model: Semiannual Product Planning Cycle 236 Always Consider Model Reuse 237 Give and Take in Building a New Model 239 Typical Model and Process Operation: Iterating 239 Keeping the Process/Model Scope Manageable and Understandable 240 Moving from Strategy to Building and Implementing Causal AI Solutions 241 Agriculture: Enhancing Crop Yield 242 Key Causal Variables 244 Creating the DAG 246 Moving from the DAG to Implementing the Causal AI Model 247 Commercial Real Estate: Valuing Warehouse Space 250 Key Causal Variables 251 Implementing the Causal AI Model 253 Video Streaming: Enhancing Content Recommendations 254 Key Causal Variables 255 Implementing the Causal AI Model 256 Healthcare: Reducing Infant Mortality 258 Key Causal Variables 259 Implementing the Causal AI Model 261 Retail: Providing Executives Actionable Information 263 Key Causal Variables 264 Implementing the Causal Model 265 Summary 267 Chapter 9 The Future of Causal AI 271 Where We Stand Today 271 Foundations of Causal AI 273 The Causal AI Journey 274 Causal AI Today 274 What's Next for Causal AI 276 Integrating Causal AI and Traditional AI 278 The Imperative for Managing Data 279 Ensembles of Data 279 Generative AI Is Emerging as a Game Changer for Causal AI 281 The Future of Causal Discovery 282 The Emergence of Causal AI Reinforcement Learning Will Accelerate Model Training 284 Causal AI as a Common Language Between Business Leaders and Data Scientists 284 The Emergence of Probabilistic Programming Languages 286 The Predictable Model Evolution Cycle 286 The Emergence of the Digital Twin 287 Improving the Ability to Understand Ground Truth 289 The Development of More Sophisticated DAGs 289 Visualizing Complex Relationships in the DAGs 290 The Merging of Causal and Traditional AI Models 291 The Future of Explainability 291 The Evolution of Responsible AI 292 Advances in Data Security and Privacy 293 Integration Will Be Between Models and Business Applications 294 Summary 295 Glossary 299 Appendix 313 Selected Resources 329 Acknowledgments 331 About the authors 335 About the contributor 339 Index 341