A Boost from AI

Intrigued by the idea of being able to predict the future, applications programmer Wayde Gilliam immersed himself in machine learning and AI. In this Q&A, the Tritonlytics Team member shares how he brought this passion for AI to the university, creating tools that help the team focus on the most important aspects of their work, including the Staff@Work Survey which runs now through September 15.

How did you become interested in Artificial Intelligence?

My background is mostly as a software engineer. Four or five years ago, I took a course on machine learning and just fell in love with it—the idea of being able to predict the future in some regard. The question I struggled with was, “How do you apply it in the workplace and deploy something people can use?”

Fortunately, I found this community and this course called fast.ai led by an AI expert and entrepreneur named Jeremy Howard. He teaches AI from a very top down approach, kind of how I would coach my kids to play baseball. For example, I wouldn’t start with the physics of bat swings or how the ball moves, I would start with the basics—the ball is going to come at you, and you’re going to swing and make contact.

I started the course, became really involved with the community and as a contributor to their deep learning library, authored an open source library, led study groups and eventually became a fast.ai community leader. 

Like many others, I’m really excited about exploring how to utilize the same kind of large language models that power tools like ChatGPT. When people use ChatGPT they think it’s AI, when in reality, it’s simply a web application that is AI-powered, and that AI power comes in the form this thing called a large language model (or LLM). At its core, LLM’s are gigantic neural networks that have been trained on a massive amount of data to predict the next word, sometimes using feedback from actual humans.

What researchers have found is that when you give these models lots of data, they can become so good at predicting what comes next that end users may start to think they’ve achieved some sense of sentience. Although it can almost feel like they are some sort of creature, it is better to think of them as a co-pilot that can help us be more efficient and automate the mundane because of what it has learned about how we talk.

When were you first able to apply this new passion to your work at the university?

I was doing the fast.ai course and contributing to a variety of open source projects with a focus on natural language processing (NLP) tasks, since that seemed likely to be very useful here.

I work with Angela Song and the Tritonlytics Team. Most of our work is around deploying surveys and providing analytics to end users to help them get a pulse of how faculty, staff, and students feel so as to take meaningful action. We started with one single survey at UC San Diego and have since expanded our services throughout campus to all the Vice Chancellor areas, Health and even externally to the Cal State Universities, universities outside California and nonprofits.

In terms of my work with the Tritonlytics Team and our surveys, I thought that the most useful application of machine learning would come in the form of working with our qualitative data, which are the comments. My thinking is that when someone spends time writing an anonymous comment, you’re getting a good sense of how they really feel. But also, it’s really hard to analyze qualitative data—it’s not as easy as being able to just look at and plot the numerical data you can get from questions where you ask someone to rate a department on a scale of one to five.

Since these qualitative questions can be so challenging and time consuming for my group, one of the first things I built was a classification model that can parse out sentiment. When the model is given survey comments, it will attempt to break those comments down by subject area. A comment may mention five different things, and we want to assess sentiment for all of them. So the model breaks that down and is trained to identify seven or eight different types of sentiment—different levels of positivity, a sense that the user might feel threatened, whether there is profanity and even if the comment might be considered nonsense. This model was initially trained on several thousand survey comments that we manually annotated. That was the baseline. As time has gone on, we have made corrections to the predicted sentiment and fed those corrections back in to model to continually improve it.

 I then turned to another area that takes a lot of time for our team. We provide a service where we will take survey comments, attempt to find themes and then report back to clients the themes and comments associated with them. It’s time consuming and also subjective. What one person thinks applies to theme A, another may attribute to theme B. It’s hard to go through 5-10,000 comments and capture the themes correctly and consistently. Given that, I built a machine learning pipeline that attempts to cluster segments of texts and predict a name for the cluster in the form of a theme.

By taking that manual labor out, we use our time to review and correct the theme names that don’t really capture the spirit of the comments. The net result is a huge time savings and the ability to better relate our findings through visualizations our team has created—like word clouds. We can then provide this service to more clients who will be better informed of how to take action in support of their staff.

With regard to the surveys, people always ask if anybody is looking at their comments and if their comments are anonymous. I hope new AI-powered tools like those we have now and are creating will make people realize that we definitely want to hear and understand their feedback.  We are committed to treating comments with the utmost respect and ensure that people’s anonymity is never jeopardized throughout the process of creating meaningful feedback for leadership. The new AI-powered tools and our humans-in-the-loop are helping all of that happen more seamlessly.  

That was all pre-Chat GPT—what are you all working on now?

There’s a lot of momentum to use the models behind ChatGPT to build sophisticated custom applications.  Here’s one example. Our clients always have questions about the data that may not be captured in the report we provide. It would be very time consuming—or maybe even impossible—to build a report with everything they could possibly ask for, so we hope to deploy in our analytics product the ability to use a chat bot that will allow clients to talk with their survey data. They’ll be able to ask questions like, What are the five things people are most unhappy with? What are the top three things we can do to improve? What are people most positive about? The chat bot will be able to provide answers on the basis of survey comments along with citations so it can be fact-checked. It is also being built so that clients can provide feedback on the chat bot responses that can in turn be used to monitor and improve the model as time goes on.

We are also working on a product to dynamically generate reports based on a user’s description of what they want to see. Currently, if someone from Tritonlytics wants to know how a specific survey did, the developers and the rest of the staff can create that, but we are looking at a way to create reports by simply having project managers talk to an application and describe what they want. The application will generate the code to create that report without having to go to a developer.

What do you think about the future of AI?

In my experience, AI works best when humans are in the loop. You automate the tedious tasks so people can focus on the important things. This makes us more efficient and improves the quality of our output.

I foresee a definite uptick in the number of AI-powered applications within our department and the university over the next three to five years. Right now, I’m convinced that most individuals don’t really understand what AI is, and don’t know whether to be afraid of it or even how it might be used. I hope to be part of clearing up those misconceptions and help others figure out how to work with AI, here at UC San Diego and beyond.