Generative AI is moving so fast that industry leaders fear falling behind if they delay acting now.

Like other technology revolutions—the Internet, smartphones, cloud technology, and IoT—that initially created much buzz, Generative AI unleashes the same frenzy.

It is nascent and susceptible to risks, yet leaders want to leverage it for greater business outcomes and benefits.

With the growing expectations around generative AI use cases, leaders want to deploy GenAI-powered applications or solutions.

Given a project’s successful needs, leaders can follow the CIO or CTO's guide to implementing Generative AI.

However, implementing a GenAI project is not the end of the game. A leader must learn too many critical things from generative AI implementation.

Lessons learned from GenAI implementation can enhance your understanding of what works best for each iteration and what to do if a system hallucinates or its performance deteriorates.

Simultaneously, what you can do better is align with the change management and bring stakeholders on the same page.

Implementation checks are essential to identify and prevent crucial challenges from impacting a project.

Let’s check what can happen after the implementation of GenAI from a real-world experience, as Workativ has demonstrated.

We also provide useful insights to avoid such challenges as you aim for a GenAI solution that performs continuously well.

By combining the power of generative AI, IT leaders can gain significant benefits for IT support.

With the IT team growing, it receives a massive volume of daily IT requests due to the influx of digital tools, such as productivity apps, collaboration tools, visual workspaces, etc.

A typical self-service bot can include automation for repetitive tasks but would be limited in extended automation capabilities.

• It can work with a handful of use cases, requiring human intervention for most queries.

• Knowledge articles reflect outdated information or are not at par with the present challenges or needs.

• IT leaders must be alert about delegating tasks to someone to update knowledge articles.

• The lack of proper skills can delay knowledge development.

Generative AI, with its LLM capabilities, can apply RAGs to its massive datasets and retrieve answers similar to user queries.

Whether the employee support queries grow, Generative AI helps improve knowledge discovery and resolve issues autonomously using LLMs.

• Generative AI continuously learns from interactions and surfaces answers even if the database encompasses no match for the query.

• With the ability to build predictive analytics, GenAI can help reduce escalation calls by providing better recommendations for issue resolution.

• Generative AI reduces the need to create knowledge occasionally with its capability to generate NLP-based content.

Generative AI can help industries eliminate the challenges for service desks to facilitate autonomous problem resolution for repetitive tasks, escalation of critical issues for human agents, and improvement of employee experience.

Workativ drives efficiency for service desk managers, helping them handle employee IT issues and requests by applying app workflow automation to generative AI.

Workativ helps one of the leading IT companies reduce the repetitive service desk challenges using generative AI-based workflows.

Generative AI-based workflows help automate multiple use cases for the following requests,

• Reset password

• Install software

• Install printer

• Unlock account

• Deactivate MFA

• Create user phone number, etc

The hands-on experience with the GenAI implementation gives exposure to some critical insights.

Let’s visualize these insights from the perspective of Workativ with our tryst with a leading IT service provider.

With Workativ, our customers do not have to do any hard work. Users can use a no-code conversational AI platform to build their preferred workflows.

We built a GenAI bot for our customer to solve their existing IT support issues.

Workativ gathered business use case data from 50+ apps and trained the GenAI platform through knowledge article uploads.

These IT support cases included information related to asset management processes, IT service uptime, user interactions, action data by agents, etc.

By enabling generative AI in the chatbot builder, the bot was tested and deployed to use LLM and surface answers using prompts.

The intent behind the IT support bot for our client was to leverage generative AI properties, automate tasks, and reduce workloads on the service desk.

However, IT support reveals certain limitations on Generative AI capabilities.

Answers were inappropriate

The GenAI bot was built to deliver ChatGPT-like responses.

However, it was not meant to surface generalized answers, and fine-tuning was not intended as it could only provide custom answers for one or two use cases.

The Workativ chatbot builder platform allows LLMs to train by uploading knowledge articles.

Later, it was observed that the self-service platform demonstrated critical challenges in surfacing wrong answers for service desk users.

For example, if a question were asked about ways to enable multi-factor authentication, it would suggest recommendations for apps that were no longer intended or part of enterprise workflows. This was one of the scenarios. There were many odd situations for users who continued to receive wrong or inappropriate answers.

It was found that the workflows pulled up information from outdated knowledge articles. These knowledge resources failed to comply with the current needs of business use cases.

Model hallucination

There were ethical considerations as well. The GenAI workflows failed to comply with the business-specific scenarios. Due to the knowledge bases reflecting old and outdated knowledge, the IT support bot provided incorrect guidance to the users. Besides, some questions without a match in knowledge bases could have answers designed with guesswork, leading to model hallucinations. A typical example of model hallucinations could be like this,

This type of suggestion did not meet the expectations of business outcomes. Also, a random suggestion could trigger confusion and the need for immediate attention from respected stakeholders.

Misalignment of document formats

One of the behind-the-scenes reasons the GenAI interface delivered inappropriate answers or gave no answers at all was that document formats for knowledge articles incurred compatibility issues with the system.

Incompatibility of document formats caused misalignment with the system and hindered the system from storing necessary information.

We gathered a couple of significant takeaways from implementing an IT support bot with generative AI capabilities for our customers.

Human error can damage great technological innovation and overshadow the potential benefits of Generative AI.

Here’s how you can measure the value of each model option a fine-tuned model, custom model, and no-code AI copilot.

Productivity

• Human-in-the-loop: Human oversight is significant in ensuring that data used for model training is always current. When the key objective is to reflect domain-specific knowledge sharing to help solve a problem, data must encompass real-world experiences from IT support about a specific business. Also, IT support can interact with industry-wide scenarios for common issues across the industry. Depending on this, users must also provide a world of knowledge via integration with correctly configured third-party LLM solutions.

Based on the observation, our customer removed outdated knowledge articles from the platform and uploaded the latest articles with use cases relevant to the present scenarios and business needs.

Also, it was recommended that any knowledge implementation should be done with AI expertise and domain knowledge.

• Elimination of hallucination: The underlying fact is that the more accurate the data, the more precise the answers are. Human oversight, while data is uploaded to the model or during the preparation of the knowledge articles, can be a good way to prevent the GenAI-based IT support bot from surfacing hallucinated answers.

Our customer ensured knowledge articles were correct and relevant to their domain-specific cases.

• Use of properly formatted documents: If the format of documents matters for a Generative AI platform that could help data ingestion, users must take care of this side. They must upload knowledge articles in preferred formats, such as PDFs, CSV files, XLS format, etc, to help the system extract knowledge correctly and surface appropriate answers.

Our customer applied documents in the right format to support system compatibility and also ensured the GenAI bot could deliver correct responses and help resolve IT issues to the maximum length.

• Data governance or AI regulations: Generative AI is known for data violation nuances. We already have many instances of data violations and security risks. The AI Act and other GDPR put into effect stricter regulations to safeguard against data use and facilitate transparency and security.

To help clients of all sizes with data protection for security and privacy and compliance with regulations, we configured knowledge bases to prevent data violation while surfacing answers. Besides, we implemented a configuration that could prevent our model from storing users’ inputs and using them to train the workflows.

• Lastly, from Workativ’s perspective, workflows can perform inappropriately due to outdated knowledge. Continuous monitoring of the system is thus significant in helping users leverage the latest information and correctly use generative AI to resolve IT support issues. This requires leaders to retain the right talent with domain expertise. There’s no doubt that no-code platforms can be leveraged to implement workflows without writing a single line of code. But, with thorough supervision from AI experts and domain knowledge, you maintain your systems and meet your users’ needs.

Generative AI implementation can be overwhelming for companies due to the lack of proper assistance from experts with domain knowledge. Deploying a Generative AI system is just a piece of a puzzle. It requires end-to-end monitoring of system performance and user experience. Workativ provides a no-code platform for Generative AI implementation. Our post-implementation support is superior to helping clients onboard and efficiently manage all processes.

Are you interested in successfully implementing your Generative AI workflows? Book a demo today with Workativ.