Using Open Source Large Language Models: The Pros And Cons
Learn About The Aspects That Make Open Source AI Components A Critical Differentiator In Your Application Stack
On November 07, Tobias Zwingmann (AI Advisor & Author) joined me on “What’s the BUZZ?” and shared how you can build Generative AI applications with open source technology. Imagine a world where complex business documents are summarized in seconds, customer inquiries are resolved instantly by chatbots, and digital content is conjured up with a few clicks. This isn't science fiction — it's the reality of AI in business today. From the cost-effective transformation of customer service bots to the rapid development of prototypes and the strategic shift towards open-source models for greater customization. The question becomes: (When) should you use open-source technology for your application? Here is what we’ve talked about…
Four Pillars Of Generative AI In Business
In the B2B sector, there are four main AI-powered applications. Summarization and Extension is all about condensing large texts into digestible highlights or expanding brief concepts into detailed content. This could apply to sales documents, marketing strategies, or any knowledge-intensive work. Search and Knowledge Retrieval is crucial for customer support chatbots, for example. It involves matching vast stored knowledge to customer inquiries. Digital Asset Creation where AI excels in generating various digital contents like videos, images, and texts, an essential tool for sales and marketing. Lastly, Code Generation is making simpler to analyze customer data and CRM tools through LLMs and code, converting natural language queries into SQL or Python code to extract valuable insights like top-selling products.
Finding The Balance Between Cost And Efficiency
Using an LLM like OpenAI’s GPT-4 involves more than just the AI's core function; it includes various tools for safety, alignment, and scalability, which can raise costs. GPT-4 is cost-effective for tasks with fewer requests, but for more extensive, automated tasks, costs can soar. It's at this stage that open-source can become an attractive alternative. For instance, in a project classifying open-text feedback from event surveys, GPT-4 performed well without errors. However, to handle continuous data from multiple events, open-source models like Llama 2, which could be fine-tuned with the data initially processed by GPT-4, could potentially offer a more cost-efficient solution.
» You can transition from off-the-shelf to more customized models by trying to fine-tune and trying to adapt models that cater to a more specific task. «
— Tobias Zwingmann
Hosting an open-source model still incurs costs for infrastructure and GPU power, but it provides a more tailored approach to specific tasks. This is a strategic shift from using broad-capability LLMs for general tasks to applying smaller, specialized models for niche problems, optimizing computational power and cost. However, enterprises often prefer off-the-shelf services for their reliability and data security assurance, which is crucial for maintaining operations and aligning with internal IT policies, despite the allure of recent open-source offerings.
The Three Layers Of The Open Source LLM Ecosystem
The open-source ecosystem for LLMs spans beyond just the models themselves, encompassing three distinct layers. The foundational layer consists of LLMs like Llama 2, Flan, and others. The second layer includes tooling resources like LangChain and LlamaIndex, and platforms like Hugging Face for model fine-tuning. This layer is dynamic, with significant ongoing developments. The third layer abstracts further, with front-end components and tools like Vercel AI and Microsoft's offerings that simplify the creation of interfaces and architectures. This ecosystem thrives on customizability, vital for bespoke use cases like managing user permissions within an enterprise. The LLM ecosystem is poised for significant growth, especially with OpenAI's introduction of customizable GPTs on DevDay, facilitating knowledge retrieval. This democratization of LLM development is currently the most promising area for short-term growth, enabling tailored solutions that meet specific organizational needs.
Summary
AI in business isn't just about automation; it's a combination of summarization, search, digital creation, and coding, transforming the B2B landscape. As needs become more complex, the transition to open-source models like Llama 2 offers a custom, cost-effective alternative. Yet, the cost to operate and maintain them as well as access to skilled resources for these nascent tools are some of the limitations to keep in mind. Beyond models, the open-source ecosystem thrives across three layers: LLMs, tooling platforms, and front-end solutions, with customizability being the key to tailored AI applications in enterprises.
What factor do open-source AI models play in your business?
Listen to this episode on the podcast: Apple Podcasts | Other platforms
Become an AI Leader
Join my bi-weekly live stream and podcast for leaders and hands-on practitioners. Each episode features a different guest who shares their AI journey and actionable insights. Learn from your peers how you can lead artificial intelligence, generative AI & automation in business with confidence.
Join us live
November 16 - Keith McCormick, Executive Data Scientist, will talk about the key roles you need to fill when building your AI team.
December 05 - Mark Stouse, CEO Proof Analytics, will discuss how managers can teach their data scientists about the business.
December 14 - Enrico Santus, Human Computation Leader, will share how you can design adaptive processes for human-AI collaboration.
Watch the latest episodes or listen to the podcast
Follow me on LinkedIn for daily posts about how you can lead AI in business with confidence. Activate notifications (🔔) and never miss an update.
Together, let’s turn hype into outcome. 👍🏻
—Andreas