Tag: ai

There has been a lot of discussion recently about companies using AI to build internal tools that replace SaaS licenses. That is interesting, but it misses a big category of software: the web apps you cannot replace.

Sometimes the constraint is technical. More often, it is not. An insurance company may require you to use its verification portal. A specialty vendor may only accept orders through a clunky ecommerce site. Or a marketplace may be where all of the demand for your product or service lives.

You can build a better internal tool, but you still have to use those sites.

The browser is the integration point

Chrome extensions have always been a way to change the experience of a site you do not control. An extension can read information from a page, add controls to it, and help guide a user through a workflow.

Historically, that was possible but often not practical. You needed to write and maintain custom code for every awkward workflow, and the payoff had to be large enough to justify it.

The latest AI models change that calculation. It is now much easier to build an extension that augments a legacy site, whether that means changing a workflow, extracting information from a page, or adding LLM capabilities directly where people are already working.

Instead of asking someone to copy information from one system into another, you can put the assistance in the browser tab where the work already happens.

A bike search on Facebook Marketplace

I recently had a good excuse to try this out. I was looking for a bike on Facebook Marketplace with a specific set of requirements. The hard part was not finding listings. It was reviewing the photos for each listing to determine whether a bike was actually a fit.

Doing that manually meant opening and reviewing dozens of listings every day. That is exactly the sort of repetitive visual task that an AI model can help with.

So I built a Chrome extension that uses OpenAI to review listing photos and flag the listings that match what I was looking for. Rather than replacing Facebook Marketplace, the extension improves the part of the Marketplace workflow that was taking the most time.

The result is not a fully autonomous bike buyer. It is a faster way to narrow down a large list of listings so I can spend my time looking at the promising ones.

Where this approach works

The Marketplace example is personal, but the pattern applies to business workflows too. Look for web-based processes where a person repeatedly has to review, classify, summarize, or move information before they can make a decision.

A Chrome extension can be a practical place to add help to:

• an insurer’s required portal
• a vendor ordering site
• a marketplace your team depends on
• an internal legacy application that is difficult to change

The goal is not necessarily to replace the site. It is to remove the tedious steps around it while keeping people in the workflow they already need to use.

See it in action

Check out the demo below to see the bike finder at work:

Watch the video on YouTube

Interested in building something similar for a workflow your team cannot avoid? Get in touch with Setfive.

A lot of information these days is just a Google search away, but there is still a surprising number of businesses out there that keep information like pricing locked behind phone lines. Oftentimes, this is deliberate, and they may do this for a variety of reasons:

• Fluctuating prices that change based on demand, inventory, or seasonality.
• Sales psychology that converts curious callers into customers.
• A competitive advantage in keeping pricing opaque to competitors.
• Personalized quotes that change based on customer need.
• Old school businesses that just never went digital.

Traditionally, to gather information from these businesses, you would need someone or even multiple people to work through an endless call list, navigating phone menu trees, waiting on hold, and manually transcribing conversations into spreadsheets. This is tedious, expensive, slow, and doesn't scale.

At Setfive, we decided to look into how we could automate this.

OpenAI Realtime API

The timing couldn't have been better. As we were exploring ways to do this, OpenAI released its Realtime API, a game-changer for voice-based AI applications. Unlike conventional text-based APIs that require separate speed-to-text and text-to-speech steps, the Realtime API combines these and enables:

• Low-latency native voice conversations.
• Natural interruptions for more human-like interactions.
• Built-in function calling for triggering actions mid-conversation.

This was an AI capable of having an actual over-the-phone conversation.

Building The Bridge

With the brain of the operation sorted, it was time to find a way to actually make phone calls. For this, we chose Twilio, a well-regarded platform for telecommunications for almost two decades.

Twilio's Media Streams API made it simple to pipe audio directly to and from the OpenAI Realtime API, creating a seamless conversation flow. The business on the other end hears a responsive customer who can handle unexpected conversational turns.

Navigating The Maze

One of the first challenges we ran into? Phone trees. You know them: "Press 1 for appointments, Press 2 to speak to a customer service representative, …" These interactive voice response (IVR) systems are designed for touch-tone input, not voice commands.

We solved this by building AI tools that can simulate DTMF (Dual-Tone Multi-Frequency) signals using Twilio's API - which required some trial and error with their callback and TwiML architecture - so that our AI can listen to menu options, simulate button presses, navigate complex multi-level menus, and find the fastest path to reach a customer service representative or a front desk.

From Conversations To Structured Data

Getting through to the right person is only half the battle. The real magic happens when our AI finally gets into a conversation. From there, we are able to extract structured information from free-flowing conversations in real time. Using carefully crafted prompts, our system can:

• Identify key information even when it's mentioned casually
• Ask clarifying questions about discrepancies in the information received
• Extract additional valuable data like availability, pricing details (first-time customer, minimum orders, ect.), and more
• Create clean, structured data ready for your database, Excel spreadsheet, or whatever else you're using.

When Nobody Answers

Here's something we didn't anticipate: businesses that rely heavily on phone communication are often too busy to answer their phones. These are often small businesses that may not have dedicated staff for handling phones or may have employees who wear multiple hats. They're not sitting by the phone waiting for calls.

This was having a real effect on our success rate, and we didn't want to make multiple calls to the same business, hoping for someone to be available. The next step was obvious: voicemail. We enhanced our system to handle a full communication cycle:

• Intelligent voicemail detection to detect when we have reached a voicemail inbox.
• Leave a natural message requesting whatever information the AI is looking for.
• Callback handling that is able to naturally continue the conversation when a business calls back.

Ready to Build?

Interested in how this can help you? Email us at contact@setfive.com to find out more or check out
our demo at voice2data.setfive.com!