A lot of information these days is just a Google search away, but there is still a surprising number of businesses out there that keep information like pricing locked behind phone lines. Oftentimes, this is deliberate, and they may do this for a variety of reasons:

• Fluctuating prices that change based on demand, inventory, or seasonality.
• Sales psychology that converts curious callers into customers.
• A competitive advantage in keeping pricing opaque to competitors.
• Personalized quotes that change based on customer need.
• Old school businesses that just never went digital.

Traditionally, to gather information from these businesses, you would need someone or even multiple people to work through an endless call list, navigating phone menu trees, waiting on hold, and manually transcribing conversations into spreadsheets. This is tedious, expensive, slow, and doesn't scale.

At Setfive, we decided to look into how we could automate this.

OpenAI Realtime API

The timing couldn't have been better. As we were exploring ways to do this, OpenAI released its Realtime API, a game-changer for voice-based AI applications. Unlike conventional text-based APIs that require separate speed-to-text and text-to-speech steps, the Realtime API combines these and enables:

• Low-latency native voice conversations.
• Natural interruptions for more human-like interactions.
• Built-in function calling for triggering actions mid-conversation.

This was an AI capable of having an actual over-the-phone conversation.

Building The Bridge

With the brain of the operation sorted, it was time to find a way to actually make phone calls. For this, we chose Twilio, a well-regarded platform for telecommunications for almost two decades.

Twilio's Media Streams API made it simple to pipe audio directly to and from the OpenAI Realtime API, creating a seamless conversation flow. The business on the other end hears a responsive customer who can handle unexpected conversational turns.

Navigating The Maze

One of the first challenges we ran into? Phone trees. You know them: "Press 1 for appointments, Press 2 to speak to a customer service representative, …" These interactive voice response (IVR) systems are designed for touch-tone input, not voice commands.

We solved this by building AI tools that can simulate DTMF (Dual-Tone Multi-Frequency) signals using Twilio's API - which required some trial and error with their callback and TwiML architecture - so that our AI can listen to menu options, simulate button presses, navigate complex multi-level menus, and find the fastest path to reach a customer service representative or a front desk.

From Conversations To Structured Data

Getting through to the right person is only half the battle. The real magic happens when our AI finally gets into a conversation. From there, we are able to extract structured information from free-flowing conversations in real time. Using carefully crafted prompts, our system can:

• Identify key information even when it's mentioned casually
• Ask clarifying questions about discrepancies in the information received
• Extract additional valuable data like availability, pricing details (first-time customer, minimum orders, ect.), and more
• Create clean, structured data ready for your database, Excel spreadsheet, or whatever else you're using.

When Nobody Answers

Here's something we didn't anticipate: businesses that rely heavily on phone communication are often too busy to answer their phones. These are often small businesses that may not have dedicated staff for handling phones or may have employees who wear multiple hats. They're not sitting by the phone waiting for calls.

This was having a real effect on our success rate, and we didn't want to make multiple calls to the same business, hoping for someone to be available. The next step was obvious: voicemail. We enhanced our system to handle a full communication cycle:

• Intelligent voicemail detection to detect when we have reached a voicemail inbox.
• Leave a natural message requesting whatever information the AI is looking for.
• Callback handling that is able to naturally continue the conversation when a business calls back.

Ready to Build?

Interested in how this can help you? Email us at contact@setfive.com to find out more or check out
our demo at voice2data.setfive.com!

Wow...it's been awhile!

A couple of weeks ago one of our clients approached us about helping them build an NFT (more on that later). In case you're not "extremely online" and don't know what web3 or NFTs are here's a quick primer.

Crypto and NFTs

As crypto currencies go Bitcoin and Ethereum are the "OG" coins. They're related projects but ultimately quite different. Ethereum differentiates itself because it enables the Ethereum Virtual Machine which is a global, distributed computing environment which uses Ethereum as payment for executing computation. Executing pieces of code, known as smart contracts, on the EVM is broadly referred to as "web3". The web3 vision is that it should be possible to transition dozens of financial businesses processes onto the blockchain by using the EVM and smart contracts to encode the rules of the processes. Think stuff like insurance, stock issuance, and even sports books.

Non-fungible tokens (NFTs) are a specific type of smart contract which encode ownership of an asset onto the Ethereum blockchain. What makes NFTs special is that because of the decentralized nature of the blockchain and the EVM its possible to freely trade NFTs and encode rules into their smart contracts. OpenSea is the defacto NFT marketplace where users can trade tokens without the original creators having to create any additional infrastructure. It's like StubHub...but anyone can sell any NFT on it and anyone can access it.

In addition, because the EVM is Turing complete its possible to enable extremely complex behaviors within the contract of an NFT. In theory, a NFT could represent ownership of any items from tickets to an event or digital collectables. But as it turns out, digital collectibles is where most of the action is today. See for example Bored Ape Yacht Club which has seen some tokens trade for upwards of $24m, Set of "Bored Ape" NFTs sells for $24.4 mln in Sotheby's online auction

OK, now that we're all caught up how does one create an NFT? There's more or less 3 steps:

1. Develop a smart contract in Solidity which implements the EIP-721: Non-Fungible Token Standard
2. Write some HTML/JS to interact with web3 via MetaMask to call your contract
3. Publish the contract to the Ethereum blockchain
4. Mint your tokens via the HTML/JS from step 2

Sounds simple enough, but how do you actually make it happen?

Here's a walk through to launch a NFT in your local test environment.

You can develop the Solidity code in any text editor. But there are some IDE options including an IntelliJ plugin and a larger list here, https://ethereum.org/en/developers/docs/ides/ It's certainly possible to write a EIP721 Solidity contract from scratch but you'll end up writing a lot of boilerplate code which will increase the surface area for bugs. A sensible alternative is to use the OpenZeppelin framework which provides you with a suite of battle tested, open source libraries to bootstrap your smart contract. Additionally, OpenZeppelin has a handful of working tutorials so that you can see a smart contract working end to end. Check out OpenSea Creatures.

After you have your contract the next piece is interacting with the blockchain to publish your contract. There's a few tools here that all interact:

1. MetaMask - MetaMask is a browser based crypto wallet and web3 provider. It allows you to store Ethereum and interact with contracts on the Ethereum blockchain. You'll use MetaMask to ultimately mint a token.
2. Ganache - Ganache is a tool which allows you to run an Ethereum blockchain on your local machine
3. Truffle - Truffle is a suite of tools which makes it easier to interact with the blockchain. You'll use Truffle to publish your contract and invoke methods within your contract.

Once you have all the tooling setup the steps you'll need to take are:

1. Setup MetaMask and note the mnemonic phrase which your keys were initialized with
2. Launch ganache with that mnemonic so that your accounts have some Ethereum
3. Use Truffle to publish your contract to your local ganache blockchain
4. Use the HTML/JS integration you wrote to invoke MetaMask to call the .mint() function in your contract

Congratulations, you just minted your first NFT in test!

The process for deploying a NFT live is effectively the same except that you'd need to buy some real Ethereum and you'd point Truffle at the live network when you publish your contract.

Hope this was helpful and we'll add more web3 related content as we continue to build solutions on it!

One of our new projects here at Setfive is a service that will allow people to create a subscription that will condense AWS product updates into a single email notification with the user’s chosen frequency. An important aspect to a product like this is a captivating sign up – we wanted to include a clean and dynamic sign up section on the website that would help to entice people to sign up and use the product.

This sign up form involves two side by side lists of ‘sources’ you would like updates about (i.e. Amazon EC2, Amazon Lambda, Amazon SNS), a textbox for your email, and a button to submit. The left side Sources are your options, which you can be adjusted via search or selecting different categories. The right side shows your selected sources – clicking a source will select it and move it to the right side, and vice versa to deselect one of your choices:

The majority of the project is handled by Symfony – Symfony is perfect for creating rather generic data entry forms made up of different input types such as textboxes, radio buttons, and select boxes. However, we wanted our sign up section to be far more dynamic than what would be easily built through Symfony’s FormBuilder.

Enter Vue.js: a JavaScript framework that can be easily integrated within a traditional web app. If you keep up to date with our Setfive blog posts, you may have seen my last blog about getting started with Vue.js. One of the key benefits of Vue.js is the ability to reuse/combine Vue components with each other and with Symfony’s forms – this allows us to reap the benefits of a dynamic/reactive Vue component as well as the automatic data validation and creation of Symfony.

The Goal:

We at Setfive love Symfony and to stay consistent, we try to use Symfony wherever possible. We wanted to reuse the ‘source select’ portion of the sign up section to allow existing users to edit their subscriptions and create new ones. However, for a registered user the create and edit subscription forms don’t require an email field and we’d instead want to immediately present ‘name’ and ‘frequency’ fields. his being the case, we knew combining our ‘source select’ Vue component with a Symfony form would be our best option – Symfony forms allow for much simpler data validation and can be displayed simply using Twig helpers.

With a combination of Symfony and Vue, we were able to build a dynamic source selecting component with Vue and allow Symfony to validate the selected sources, the name, and the frequency automatically without any extra work.

The Solution:

The first thing we needed to do was split our existing ‘source select’ component up so that the double list selector is independent from the other fields on the sign up form. Fortunately, it is simple to create parent and child Vue components and pass data from child to parent. This is done through Event Emitting: when a source is selected in the child component (source select), that ‘event’ and its data is emitted to the parent component (form composed of source select + email field and submit button).

It is a bit more complicated to synchronize this data with a Symfony form. To solve this problem, a few steps were needed.

First, we had to see what a form would look like if we did this without Vue – in other words, if we created a form and allowed you to use checkboxes to select your sources, what would the HTML elements of the individual sources look like when selected/not selected?

<input type="checkbox" id="subscription_edit_sources_5" name="subscription_edit[sources][]" class="form-check-input" value="5" checked="checked">

Our Symfony Form type ended up looking like:

<?php

namespace App\Form\Type;

use App\Entity\Source;
use App\Entity\Subscription;
use Doctrine\ORM\EntityRepository;
use Symfony\Bridge\Doctrine\Form\Type\EntityType;
use Symfony\Component\Form\AbstractType;
use Symfony\Component\Form\Extension\Core\Type\ChoiceType;
use Symfony\Component\Form\Extension\Core\Type\SubmitType;
use Symfony\Component\Form\FormBuilderInterface;
use Symfony\Component\OptionsResolver\OptionsResolver;

class SubscriptionEditType extends AbstractType
{
    public function buildForm(FormBuilderInterface $builder, array $options)
    {
        $builder
            ->add('name')
            ->add('frequency', ChoiceType::class, ["choices" => array_flip(Subscription::$FREQUENCIES)])
            ->add('sources', EntityType::class, [
                'class' => Source::class,
                'multiple' => true,
                'expanded' => true,
                'choice_label' => 'name'])
            ->add('submit', SubmitType::class, ['label' => "Save"])
        ;
    }

    public function configureOptions(OptionsResolver $resolver)
    {
        $resolver->setDefaults([
            'data_class' => Subscription::class,
        ]);
    }
}

Next, our Subscription form needs to include that form element (sources), but not actually render it on screen. Via Twig:

{% do form.sources.setRendered %}

This way, ‘sources’ is a form element whose data will be submitted, but not displayed via Twig.

Finally, we need to handle the logic of sources being selected and deselected. By tracking the ID of each source, we can create hidden HTML elements containing the exact same data that would be present if we were rendering the form entirely with Symfony and Twig.

When a source is selected, our parent component receives that data and we create the corresponding element, without displaying it (class=”d-none”):

$("#subscription-form")
.append($('<input id="source_'+source.id+'" type=text class="d-none"/>')
.attr('name', 'subscription_edit[sources][]')
.val(source.id)
.prop('checked', true));

When a source is deselected, we simply delete that element:

$("#source_"+source.id).remove();

Once a user hits submit, the form data containing those hidden input elements is compiled and the ‘sources’ form element mentioned above will now contain a list of the source IDs. Behind the scenes, Symfony converts those IDs to their corresponding Source object and Voila! Your subscription now contains the sources you chose!

Have any questions or feedback? Let me know in the comments!

Software engineering teams face many struggles, from the small problems during feature development, “Oh shit there’s no way for a user to change their last name…“ or “How do I update a client’s old web page to autoplay in screen video with scrolling?” to much more dire problems, like having to fix a catastrophic failure real time (see the case study below on one company’s crash and burn). Or that last push with a bug fix only had a test for one failure mode, but doesn’t capture or reproduce all possible modes. Can anyone be sure if it really fixed the bug?

What is a Checklist

A checklist is a memory aid – a tool for eliminating failure. A “to-do” list jotted on the back of an envelope is an informal reminder of a list of actions that need to take place, maybe prioritized by urgency; a more formal checklist might have hierarchies for lists within lists in checkbox bullet form; an automated checklist could include fail safes to prevent a user from progressing in a script before a required step is completed. The more automation is possible, the better since it prevents the possibility of human error.

A Knight’s Fall

What happens when humans are left to their own devices to do unfamiliar or infrequent tasks? In the context of DevOps, we have the tragic tale of Knight Capital Group (link to story), a 400 million dollar market making company that went bankrupt in just “45 minutes of hell.” In 2012, Knight represented a significant portion of the market share on NYSE and NASDAQ, and their Electronic Trading Group (ETG) dealt in high volume trades – billions – every day. All was well until NYSE planned a launch of new program, prompting Knight to update their algorithm for routing orders. As part of this update, one fatal mistake occurred at the level of manual deployment.

“During the deployment of the new code, […] one of Knight’s technicians did not copy the new code to one of the eight SMARS computer servers. Knight did not have a second technician review this deployment and no one at Knight realized that the Power Peg code had not been removed from the eighth server, nor the new RLP code added. Knight had no written procedures that required such a review.“

SEC Filing | Release No. 70694 | October 16, 2013

The root cause was a malexecuted deployment, but taking a step back, you might say they put undue responsibility on the engineers deploying it. The ethical onus is greater on any entity whose dealings have such immediate and high-visibility leverage on the global economy as a whole. Knight’s failure to include automated processes as part of the software update, or for that matter, any documentation (EMERGENCY C/L, anyone?) on how respond to potential failure, belies any assumed responsibility for the weight of their role.

What can checklists do and what can’t they do?

In The Checklist Manifesto: How to Get Things Right, accomplished surgeon and New York Times bestselling author Atul Gawande conveys the importance of the simplest of any tool in a surgeon’s tool belt: the checklist. Checklists are used in fields ranging from disaster recovery and military to medicine and business. For high stress or time sensitive tasks, like emergency procedures, critical thinking and decision making skills may be impaired by the stress of the situation. Checklists are absolutely vital in these situations. How about for other, less important situations? Use a checklist wherever it seems appropriate, and don’t use one where it’s not.

Do you plan on implementing checklists at your workplace for preventing potential failures? What pitfalls have you experienced with the use of checklists, or lack of? Leave a comment.

Have you ever tried pitching an upgrade to management? Odds are, you probably didn’t find yourself walking away with a blank check. Maybe you’re a network administrator for a real estate company whose boss doesn’t understand why the cheaper network infrastructure isn’t always the best option for scalability; or maybe you need to request an upgrade for an application that takes up a significant amount of your time every day to troubleshoot because it’s incompatible with other operating systems. The conversation goes something like:

You: “Boss, we really need to upgrade [xyz] software package.”

Them: “Why do we need the upgrade? If it ain’t broke, don’t fix it.”

Your: “Well, it’s creating a number of issues for our team. The manufacturer no longer supports the version we use, because it’s been obsolete for 10 years. Whenever an issue comes up we have to come up with a workaround.”

Them: “How much is the upgrade?”

You: “It’ll be $ X for a shared license for all team members.”

Them: “I just don’t think we don’t have the money in the budget for that kind of upgrade. We have a lot more pressing projects requiring capital right now, and I can’t see us justifying that expense to our board.”

Such a request may not be well received because of difference in perception of the situation –of the cost-reward assessment of the solution. The management team may not speak the same language, so to speak, as the technical support or engineers, so it’s crucial to put the request into terms they will understand and listen to. Better yet, frame that request as an offer.

Here’s three ways here that you can sell to that point, in language even your boss can understand.

1. Security

Technology has never had an obsolescence rate as fast as it is today. Failure to keep up to date is not just a matter of having the best and newest techy toys, though; it can lead to a security breach of personal information (like the Target PIN data breach of 2013), stolen identities and stolen money.

Cyber security expenses are perhaps the hardest sell to make, considering failure to upgrade presents a latent risk rather than an active one. It works until it doesn’t. Earlier in 2020, the stock market witnessed a reactionary boost in security spending in companies like FireEye, after celebrities like Elon Musk’s Twitter accounts were hacked.

As an engineer pitching an upgrade to management, convey the risk of not getting it and the potential fallout.

2. Developer productivity

A software version upgrade can be money in the bank if it saves man hours by making tasks less labor intensive and more efficient. Use terms like “faster,” “leaner,” or the military favorite “force multiplier.” If you’ve got estimates on time allotted to a given project that can be broken down into hourly direct labor savings, that’s always a great selling point.

3. Hiring and Retention

The success of any project depends not just on the tools, but on the people using those tools; to a large degree, the more cutting edge your tools are, the more cutting edge the people in your employ will be. When it comes to hiring and retaining employees, one deciding factor will surely be how modern your operating environment is.

If your team uses obsolete tools, it may even be more difficult to find someone with that skill. If you use Python 2 instead of Python 3, the syntax and many features are quite different, but all modern users are taught the most recent version, so it will present a small challenge to hire someone who’s willing to use (or learn) an obsolete version of that language.

Whatever the case, it’s safe to say meeting the bottom line is among any company’s top priorities, when it comes to spending. The more you can appeal to that end and sell a tangible ROI for the cost of the upgrade, the more likely you are to hear a ‘yes.’

Another tip is to present multiple options: a good, better, and best option with #1 being the most expensive. People are often more likely to choose to do something when it is presented as one of multiple options than alone.