[Musing] Tracking the Right Metrics Key to Manifesting AI’s Impact on Workforce

This musing is part of an ongoing collection of articles disseminating academic research and industry conversations on AI’s broad societal impact.

Here’s the takeaway:

As academic and applied researchers, we don’t just study or track social reality. The metrics we create and use also shape and influence that reality. We have the expertise and responsibility to lean into conversations shaping the design of target metrics during social transitions. In the case of AI’s impact on workforce, the focus on “AI capability” as the only desired metric by frontier labs reinforces their ever-growing interest to prove, at all costs, they have the most capable AI. But AI could have a much more positive impact and could lead to creation of new work opportunities if we can influence a shift to track the right metrics, such as organizational mission success, emerging demand for work, and AI risk.

A prevalent narrative around AI’s impact on the knowledge workforce goes like this. If AI can complete X% of tasks, then the workforce will be reduced by Y% in Z years.

Is that really the inevitable future? Do we knowledge professionals really have no agency or power to affect this future?

Not necessarily. Two papers we recently came across through UC Berkeley’s ongoing Colloquium in Law, Philosophy & Political Theory highlight three rather actionable opportunities.

The first opportunity concerns the development of good metrics to measure long-term impact and mission success. Case in point is the recent surprising staff growth at Georgia State University (GSU) after engaging AI tools.

GSU introduced a new data and academic alert platform beginning in 2011. The system tracks a list of student outcome indicators and proactively alerts academic advising staff when a student is anticipated to need additional support. In the years following the initiative, GSU’s advising staff more than doubled with 42 additional advisors hired and a new centralized advising center was established.

The working hypothesis of this surprising workforce growth after technology use is this: Technology was viewed by the organization as a tool to improve student success and institution-wide financial health. Labor cost wasn’t a target metric; organizational mission success was. So when technology was able to identify the work in need at scale, advising roles were created and reimagined to meet the need. It’s about the mission and playing the long game.

Many of us applied researchers, especially evaluators, have had the experience to help a mission-driven organization stay focused on what’s important. Moving forward, this will not just be a good-to-have; it will be our moral obligation to do so.

GSU’s story also accentuates the importance to track shifting demand for work rather than only loss of work.

Say the current cost of a human doing task X is $500. If the cost of having it done by AI is predictably lower than that, AI will very likely be favored, and job displacement will likely occur. That narrative on job displacement is what we have been focusing on so far. However, it overlooks an important possibility - the shift in what “work” entails.

Assuming AI becomes a reliably affordable solution, its lower cost will likely attract more investment in AI adoption, which in turn means more demand for human labor in infrastructure and to implement the adoption process. (Our Work of The Future Report talks more about this and makes a few additional predictions.)

As the adoption process widens and deepens, AI will likely provide accessible solutions to tasks that used to be too expensive to tackle on a routine basis. As costly solutions become accessible, “work” changes with more opportunities uncovered. For example, if personalized DNA-based disease detection is as affordable as a COVID test kit, we might experience a major shift in healthcare from diagnostics to preventative medicine. Numerous work opportunities in preventive care will be created.

Of course, there is another, less equitable possibility hidden in there. With more funds chasing sectors with higher return on investment (ROI), important tasks with lower ROI might be under-delivered. As is the case today, these socially important work areas - such as caregiving and grassroots organizing - may have to continue to rely on unpaid or underpaid labor such as family and volunteers.

There are other possibilities. But unless we are tracking these additional possibilities alongside job displacement, we are missing a great opportunity to bring about the more socially beneficial outcomes.

Some of you might be questioning how creating more jobs in building AI infrastructure - such as data centers - is benefiting the world. Indeed, I agree. That brings us to the third opportunity for us to exert an impact during this transition - defining and measuring AI’s risk.

Like any technology, AI has the potential to have broad, potentially negative impacts on the sustainable growth of our society. That risk doesn’t just concern communities, it concerns business adopters as well.

Recall for AI to even start to have any impact on job displacement, the assumption is that it needs to be “reliably affordable”. Currently, AI tools come with a price tag. But for business decision-makers, that price tag doesn’t include the cost of risk. It’s way too early to fully understand the risk of AI. Therefore, few can confidently state its full cost. This could explain why only a few companies with the resources and interest to train their own AI with proprietary data actually implemented it. They’re still waiting for the risk assessment. Appeal is one thing; paying a high ticket for it is another.

This opens the door to discussions on what AI risk is and how we measure it. Is it the direct financial liability brought to businesses? Is it the loss of trust among its customers and shareholders? Is it profit volatility down the road if the consumer economy is hit hard by stagnant real purchasing power? Is it long-term unsustainability that might shake every aspect of business operation?

There isn’t a set way to define AI risk yet. But as researchers and evaluators, we have the expertise and responsibility to change that.

We live in a time of potential persisting social transformations. Uncertainty often feels scary. I hope this musing restores a sense of agency and empowerment. Only a negligible fraction of humans throughout our collective history have had the opportunities to shape our societal trajectory. We are lucky to be among them.

Here are a lot of actions you can take today:

• Talk with your community about how they can help shape the conversations about AI’s impact on workforce and beyond

• Ask your government officials, at the federal, state, and local levels, how they plan to define and track AI’s impact

• Join our events at The Future Professional Initiative to meet like-minded folks to start to build collective power

• Host a webinar with us to highlight your learnings, success stories, and experiences

Looking forward to seeing what the future holds.