We are currently in the midst of a boom era for Generative AI, and the landscape is changing almost daily. For more details, please refer to this blog post. The possibilities seem endless!
However, this is also an increasingly confusing time for builders. After spending months fine-tuning GPT-3, you may wake up to find that the model is now irrelevant due to the availability of GPT-4. Similarly, while working on building your workflow, you may discover that AutoGPT has taken over your niche.
In this fast-changing era, it's crucial to build a solid foundation that allows you to leverage the benefits of the generative AI revolution.
For the front-end system, one of the key challenges is to be flexible enough to adapt to different distribution channels such as the Chrome Web Store, Apple App Store, websites, and Slack App. Each channel has its own front-end UI development guideline. However, as the current AI era represents a paradigm shift, it is unclear which channel will eventually emerge as the winner. Therefore, it is essential to be able to experiment quickly at this stage. This often means revamping the front-end UI every few months. It would be a huge waste of effort if a team had to start from scratch each time they need to prioritize a new distribution channel.
The critical challenge for AI systems is to design an architecture that is agnostic to changes in the generative AI platform. For an overview of the generative AI platform, please see this A16z blog.
Accuracy has been a major issue for Large Language Models (LLMs), which are the workhorses of generative AI. LLMs from Google, OpenAI, and Microsoft all suffer from "hallucinations," in which the AI confidently generates fake statements. Accuracy is a significant obstacle to applying LLMs to high-value use cases, such as enterprise applications. Any company working in the generative AI field must have a systematic way to improve accuracy of their AI system.
Two capabilities are required to address accuracy problems.
Fast user feedback and adaptive learning. Users should have the ability to report inaccuracies, and the system should be able to quickly adapt to feedback.
System to Ensure Accuracy: An AI system should adhere to certain rules or guarantees. One important rule is that the output should never be inappropriate. Additionally, due to the probabilistic nature of LLMs, it is necessary to build a software stack on top of them to ensure accuracy.
Generative AI can be both expensive and slow. While APIs like OpenAI and open source models like LLaMA have made it easier to build generative AI systems, it remains challenging to build a scalable back-end system. The fast-changing landscape of generative AI makes this particularly difficult.
If you choose to use an API, the latency is often very high. Therefore, you need to design a system that can accommodate this latency. On the other hand, if you choose to use open-source models, the accuracy is likely to be worse. In this case, you need to design a system that can improve accuracy, for example, by using certain types of fine-tuning. Additionally, you need to design your system to tune efficiently because the base models change very rapidly.
Many API and open-source models have limits on the maximum amount of text you can send to the API, known as the token limit. To ensure that your system works within this limit and does not negatively impact your model's performance, you need to design it accordingly.
Many API providers specify a maximum quota for API calls. You will need to work within this requirement by minimizing the number of API calls made.
Tight-knit development workflow
The typical product development cycle involves the following steps: user feedback collection -> feature prioritization -> development -> feature impact analysis. It is crucial to accelerate this cycle and make it as continuous as possible.
When collecting user feedback, you may have a separate customer team responsible for its collection. However, it is crucial for your engineers to also listen to user feedback. This requires having a development team that is highly focused on the needs of users.
Having a systematic way to identify important features that will benefit the business is crucial for feature prioritization. It's essential to align the entire team with a single metric, known as the north-star metric. Prioritization should be done on a daily basis, rather than quarterly.
For software development, it is essential to be able to ship code as quickly as possible. Both technology capabilities and market needs are evolving rapidly. To find a product-market fit, you need to move faster than both. This means you need to be able to ship code continuously. Investment in software development lifecycle (SDLC) tools and continuous integration/continuous delivery (CI/CD) systems is crucial.
For feature impact analysis, it is important to collect user data as early as possible. This allows you to analyze user patterns and identify the top use cases. It is also worthwhile to invest in the business intelligence stack early on.