The new rules are yet unwritten: Inside Notre Dame’s Computational Methods Workshop

Computational Methods Workshop

Ask Nicholas Berente what’s keeping management researchers up at night, and he doesn’t hesitate.

Across the board, it’s the effort to ethically and appropriately wrangle AI tools for their research.

“This is the issue in management research right now,” said Berente, the James H. Sweeny III and Alicia Sweeny Collegiate Professor of IT, Analytics, and Operations at the University of Notre Dame’s Mendoza College of Business.

For decades, the field ran on well-worn methodological tracks, such as econometric regression, behavioral experiments and surveys, and qualitative case studies. Each had its own conventions for establishing strong findings and inferences that could be trusted.

Generative AI and the broader wave of computational tools haven’t given researchers a new track to run on, but they’re blurring the lines between the old ones.

And nobody, Berente said, has yet written the rules for what counts as valid, rigorous scholarship in this new landscape.

That gap in the rulebook is what brought more than 80 researchers to the Mendoza College of Business at the University of Notre Dame on June 2 for the Computational Methods in Management Workshop. Berente and Ahmed Abbasi (the Joe and Jane Giovanini Professor of IT, Analytics, and Operations, Director of the Lucy Family Institute for Data and Society, and Co-Director of the Human-Centered Analytics Lab) organized the one-day event with support from three of the premier associations in the management field: the Academy of Management, INFORMS and the Association for Information Systems.

The workshop came together in a matter of months, Berente said. Once invitations were sent, speakers said yes almost immediately, and a call for papers drew roughly 80 submissions from scholars. The response was, Berente said, “breathtaking.” It suggests the field is hungry for exactly this conversation.

Humans Required in the Loop

The workshop’s speakers were focused on today’s real research problems and opportunities in research methods. They came to show, in granular detail, what AI-backed tools can already do. In doing so, they ended up making an unexpectedly clarifying case for why human researchers aren’t going anywhere anytime soon.

Computational Methods Workshop

Suprateek Sarker, an information systems scholar from the University of Virginia who serves as editor-in-chief of Information Systems Research, delivered the evening keynote. A qualitative researcher by training, Sarker described a project in which he never interviewed a single human subject. Instead, he used generative AI to create fictional personas, “interviewed” those personas with the same tool and then used AI again to analyze the transcripts. It’s the kind of workflow that would have been unthinkable in qualitative research a decade ago — and it worked, in the sense that it produced a complete study.

But Sarker himself was blunt about the ceiling on that work: the result, he said, was not compelling enough to be accepted into the top-tier journal where he serves as editor. The tools could execute a research process, but they couldn’t replicate the judgment that makes a contribution matter.

A second presentation drove the same point home from a different angle. João Sedoc, assistant professor of technology, operations and statistics at NYU built an AI agent designed to imitate a well-known finance professor’s approach to valuing organizations. He fed his agent the professor’s papers, his recorded lectures and transcripts of his classes.

“We have trained this model about as well as you’re going to train a model,” Berente said of the effort. And yet the agent’s output topped out at the level of “a mediocre student,” Berente added. The quality of its writing and analysis was nowhere near that of the professor it was built to emulate, and not even as strong as his best graduate students.

For Berente, both examples point to the same conclusion: When researchers use these tools as a substitute for scholarship rather than a supplement to it, they get mediocre results, not work that moves a field forward. What’s missing isn’t computational power, it’s a human’s ability to contribute to theory, which Berente defined simply as “the articulation of the cumulative body of knowledge” in a domain.

Theory is the accumulated understanding that allows a scholar to recognize whether a finding is actually new or just dressed up to look that way. An AI system can be fed enormous amounts of raw material, but it doesn’t build upon theory the way an expert does.

A Need for New Consensus

Computational Methods Workshop

The day’s discussions left Berente reflecting on two open problems the field still needs to work out. One is about the data itself: Generative tools can produce synthetic data that’s useful for some purposes, but that data reflects whatever flawed material the underlying model was trained on — and researchers don’t yet have agreed-upon criteria for when generated data is defensible in a study and when it isn’t.

The other concerns analysis. Many computational tools now embed their own built-in analytical methods, often in ways that are hard for a researcher to fully inspect. This raises a harder question: How do you establish that a result is valid when the process that produced it is partly a black box?

“We don’t have a really good methodological script for either problem yet,” Berente said.

Notably, even the workshop’s two co-organizers approach this terrain differently. Abbasi’s expertise is in human-centered computational design: using existing behavioral and organizational theory to build artifacts and systems, work that sometimes contributes back to theory but more often draws on it. Berente’s focus, which he calls “computationally intensive theory construction,” runs the other direction, and involves using computational tools to extend and build new theory itself.

Both have special journal issues in the works this year on related questions: Abbasi’s, in Information Systems Research, on generative tools and research methodology. Berente’s, in MIS Quarterly, on the ethics, regulation and policy questions AI raises for the field.

A Room Full of Young Scholars, and a Reason for Optimism

The afternoon’s Paper Development Workshop (PDW) offered a case for optimism in the evolving landscape of management research.

Organized by Bei Yan, assistant professor at Stevens Institute of Business, and Romilla Syed, associate professor of information systems UMass Boston, the PDW gathered doctoral students, junior faculty and research teams from around the country who had submitted their in-progress work for structured feedback. Journal editors sat as mentors at roundtables, and attendees worked through research summaries together.

Berente admitted he’d approached the session with some of the wariness common among senior scholars watching AI tools spread through their field: a quiet worry about what happens to rigor when powerful new tools land in inexperienced hands. Instead, he came away energized and hopeful.

The questions the younger researchers brought were, in his words, genuinely difficult. (“Ethical questions,” “hardcore computational questions,” “applied questions,” he said.) None of them had easy answers.

“I left the conference optimistic,” he said, adding that he doesn’t see any of these tools shutting down the profession so much as opening up new avenues that didn’t exist before.

Abbasi notes, however, that AI tools are advancing on a near-daily basis.

“The limitations we see today will no longer be limitations tomorrow,” Abbasi said. “Many believe that these tools will quickly replace many research activities. It’s up to leaders like those who attended the workshop to investigate and establish when and how these tools should be used in the research endeavor.”

Computational Methods Workshop