The Hidden Risk in ‘Set It and Forget It’ AI
The promise of AI-powered prescreen marketing is seductive: upload your criteria, let the system match bureau data to qualified prospects, and watch loan applications roll in. But new research from MIT Sloan Management Review and Boston Consulting Group reveals a critical flaw in this “set it and forget it” approach—one that community financial institutions cannot afford to ignore.
What 84% of AI Experts Want You to Understand
In their fifth annual study of responsible AI implementation, MIT SMR and BCG assembled an international panel of AI academics and practitioners to assess what responsible AI actually demands. The headline finding: 84% of experts agree that responsible AI efforts fail if they don’t cultivate human experts who can verify AI solutions.[1]
But here’s where the research gets interesting—and directly relevant to prescreen strategy. The experts defined “verification” far more expansively than a simple thumbs-up on final outputs. As the researchers explain, verification encompasses “the work of applying human judgment across an AI system’s life cycle, interpreting context, designing tests, auditing workflows, setting thresholds, weighing when AI should not be relied on at all, and carrying the accountability that machines cannot.”[1]
In other words, responsible AI isn’t about having a loan officer glance at a prescreen list before it goes out the door. It’s about embedding human expertise into every stage of how that list gets built in the first place.
Why Context Cannot Be Automated
The MIT SMR research surfaces a theme that should resonate with every community FI leader: context is irreducibly human.
TÜV AI.Lab CEO Franziska Weindauer notes in the study that “AI solutions operate within complex real-world contexts, and human experts are essential to interpret results, detect failures, and ensure that systems function as intended.”[1] OdeseIA president Idoia Salazar puts it more directly: “Not everything is translated into data, such as context in a specific situation.”[1]
Consider what this means for prescreen marketing:
- Credit thresholds aren’t universal. A 680 FICO minimum might be appropriate for an auto loan campaign in a stable employment market, but too aggressive in a region experiencing layoffs at a major employer. That’s contextual judgment no algorithm possesses.
- Offer structures require local knowledge. Bureau data can identify debt-to-income ratios, but it can’t tell you that a particular segment of your market is disproportionately affected by seasonal income variation—knowledge your lending team has accumulated over years.
- Exclusion decisions demand accountability. Sometimes a segment qualifies on paper but shouldn’t receive offers due to factors the model can’t see. Humans must own those decisions and the reasoning behind them, including regulatory requirements.
As GovLab chief research and development officer Stefaan Verhulst explains in the research, “Many of the most significant risks of AI are societal rather than technical, such as misalignment with public values, harmful impacts on vulnerable groups, or inappropriate deployment contexts.”[1] For community FIs, those aren’t abstract concerns—they’re reputational risks tied to specific neighborhoods, member relationships, and regulatory expectations.
Verification as Connective Tissue, Not Checkpoint
The MIT SMR researchers offer a reframe that should shape how community FIs evaluate any AI-powered prescreen solution: “Verification is not a final checkpoint but the connective tissue of responsible AI, encompassing the design, oversight, and accountability that organizations need to scale alongside the systems themselves.”[1]
Applied to prescreen marketing, this means human expertise should be embedded at multiple stages:
- Design: Which credit attributes matter most for your specific loan products and risk appetite? What exclusion criteria reflect your institution’s values and regulatory posture?
- Threshold-setting: Where should cutoffs land—and how should they adjust based on campaign goals, portfolio composition, or economic shifts?
- Ongoing governance: Who reviews campaign performance, identifies unexpected patterns, and decides when parameters need adjustment?
- Accountability: When something goes wrong—an offer reaches someone it shouldn’t have, or a high-potential segment gets excluded—who owns the explanation and the fix?
None of these functions disappear because you’ve adopted AI. They become more important, because the speed and scale of AI-powered campaigns amplify both the benefits of good judgment and the costs of its absence.
The Competitive Advantage of Human-Centered AI
Megabanks can afford to treat prescreen as a pure volume game—blast millions of offers and let response rates sort themselves out. Community banks and credit unions cannot. Your margin for error is smaller, your member relationships more personal, and your reputation more locally concentrated.
That constraint is also your advantage. The same deep contextual knowledge that makes “set it and forget it” AI risky for community FIs is precisely what makes human-centered AI powerful. You know your markets. You know your members. You have lending expertise that reflects decades of local experience.
The research confirms what effective community FI leaders already sense: responsible AI doesn’t mean removing humans from the equation. It means designing systems that make human judgment more scalable, more consistent, and more impactful—while keeping accountability exactly where it belongs.
In prescreen marketing, that’s the difference between generic automation and genuine competitive differentiation.
