The Future Of AI: The Diverse Skills Needed To Solve Complex Business Challenges
Why Blending Traditional Machine Learning With Generative AI Remains A Key Pillar For Successful AI Teams In Business
On November 16, Keith McCormick (Executive Data Scientist, Pandata) joined me on “What’s the BUZZ?” and shared the key skills to build on your AI team. AI is transitioning beyond basic prompt engineering and requires a more comprehensive skill set. Despite the hype surrounding LLMs, traditional machine learning techniques, such as decision trees and random forests, maintain their significance, underscoring the fact that AI's roots are as vital as its burgeoning branches. The importance of understanding client needs, aligning AI solutions with business objectives, and fostering effective collaboration across organizational departments remains a central theme. But how can AI leaders best shape their team going into 2024? Here is what we’ve talked about…
AI Skills Beyond Prompt Engineering And The Hype
In the realm of Generative AI, there's a growing consensus that prompt engineering alone isn't sufficient, especially for professionals in AI consultancy. This perspective is becoming increasingly relevant in journalism and marketing research. Sole reliance on prompt engineering is too limited, especially for tasks that involve both structured and unstructured data, like form processing. The initial excitement around prompt engineering as a standalone skill is waning. What's needed instead is a diverse skill set that includes managing different data types and integrating AI solutions into existing pipelines and APIs. AI professionals should be equipped with a wide array of skills to effectively handle various types of data and AI applications.
Bridging The Gap From Technical Solutions To Business Outcomes
Working with AI demands a deep understanding of your stakeholders’ needs. They often approach AI leaders with a general idea, like wanting to use LLMs, but without a specific problem or question in mind. The key skill is engaging in a dialogue to pinpoint what they're actually trying to achieve. This process is crucial, and it's something new data scientists usually learn by observing and participating in several projects. It’s not just about solving a technical problem; it’s about aligning solutions with business problems that have financial implications. The goal is to ensure projects are financially viable and don't drag on indefinitely. A common misconception is that business stakeholders always know exactly what they need. In reality, it’s often a collaborative process to define the problem accurately.
» The most important skill to have at the team is this dialogue with a client or internal customer about what exactly they're trying to do. «
— Keith McCormick
LLMs are a prime example where technical solutions need to be tied to business metrics. Simply implementing a chatbot or document search powered by generative AI isn’t enough. It’s necessary to identify how these solutions contribute to cost savings or other business goals. A good example is upgrading a chatbot with generative AI, which, while technically appealing, must also be justified in terms of cost-effectiveness. Without this alignment, it's impossible to measure the success of a project.
The Need For Diverse Skill Sets On An AI Team
The discussion on AI's evolution highlights how some perceive traditional machine learning skills, like decision trees and random forests, as outdated. However, these techniques remain vital. Despite the rise of large language models and chat engineering, industries still rely heavily on supervised machine learning for tasks such as detecting insurance fraud, loan defaults, and predictive maintenance, like in IoT sensor projects. The necessity for diverse skill sets in AI teams is clear. While knowledge of LLMs is crucial, it's part of a broader spectrum of natural language processing. With advances in computer vision, this area has become a specialty in itself. AI teams need experts in both these domains, likely cross-trained, but with distinct specializations. Additionally, proficiency in traditional, effective supervised machine learning is essential.
Architecting an AI project requires a combination of various skills. Team members should ideally have overlapping expertise across different areas, including unsupervised learning. The integration of new techniques, like word embeddings, with older methods such as cluster analysis and factor analysis, is crucial. For instance, in natural language processing, it’s not just about text analytics; it’s about integrating these analytics into predictive models and understanding the trends they reveal. AI professionals must blend new advancements with established techniques, ensuring a comprehensive approach to AI solutions.
The Importance Of Team Collaboration In AI Projects
A key consideration in any AI project is the interaction between different teams within an organization, like data science and the business. It's crucial to avoid internal competition and lack of communication, as this can lead to organizational resistance, which is often the main barrier to deploying AI models. The goal of building and deploying AI models involves more than just technical prowess; it requires a deep understanding of the organizational context and the ability to collaborate effectively with various internal or external stakeholders. This approach ensures that the models deliver real value and are maintainable over time. AI deployment in an organizational setting is as much about technical ability as it is about understanding the unique needs and dynamics of different departments, ensuring seamless integration and collaboration.
Summary
The AI and data science landscape is evolving. There's a distinction between pure prompt engineering and broader AI applications. AI professionals need to have a deeper understanding of natural language processing and other aspects of AI as well. Understanding client needs and aligning AI projects with business objectives is a key skill and objective that new data scientists should learn quickly. Traditional machine learning techniques remain crucial, especially in industries dealing with structured data. The integration of both new and old techniques is essential in various AI applications. But, organizational resistance to AI projects appears often due to a lack of communication and collaboration between departments. Cooperation between teams can help avoid conflicts and ensure successful AI deployment. These insights demonstrate the multifaceted nature of AI and data science, underscoring the importance of a holistic approach that encompasses technical skills, business acumen, and interdepartmental collaboration.
In your experience, what skills are most crucial for a successful AI team?
Listen to this episode on the podcast: Apple Podcasts | Other platforms
Become an AI Leader
Join my bi-weekly live stream and podcast for leaders and hands-on practitioners. Each episode features a different guest who shares their AI journey and actionable insights. Learn from your peers how you can lead artificial intelligence, generative AI & automation in business with confidence.
Join us live
November 28 - Mark Stouse, CEO Proof Analytics, will discuss how managers can teach their data scientists about the business.
December 14 - Enrico Santus, Human Computation Leader, will share how you can design adaptive processes for human-AI collaboration.
Watch the latest episodes or listen to the podcast
Follow me on LinkedIn for daily posts about how you can lead AI in business with confidence. Activate notifications (🔔) and never miss an update.
Together, let’s turn hype into outcome. 👍🏻
—Andreas
That was a good discussion, Andreas. What are your thoughts on whether the professional interfacing with teams or clients who need AI deployments themselves need to have an AI/technical background? Can someone be adept in consulting without having the engineering depth and still be effective?
Great question, Matt! I think it depends on the scope of the project and what an individual is able to cover (or says that they are able to). If it’s about selecting the best model/ algorithm or building a model yourself, I’d convinced that consultants will need to have technical skills. If it is about identifying opportunities for leveraging AI within the business or within a process, it is much more important to select the right process/ step/ KPIs to start with and to have an understanding what to leverage AI for (rather than the technology/ components of the stack underneath). If you are a consultant who has expertise in both technology AND business, you have the perfect skillset and can go either way.