AI code generation is one of the most useful things to happen to software development in a long time. We use it. It gets people from an idea to a working application much faster than they could have a few years ago.

That is a big deal. It is also not the same thing as getting from an idea to a well-operated application.

This week, one of our clients’ Snowflake cost alerts went off. A new application had spent more than $4,000 in a couple of days. The application had been built with Claude and it was doing what its owner intended it to do. The problem was how it was doing it: repeatedly polling Snowflake with a larger warehouse than the work required.

The code worked. The bill did too.

The alert was the important part

The client had anomaly alerts in place long before this application existed. Over years of normal use, those alerts had established a useful picture of what ordinary compute usage looked like. When the new application’s usage departed from that pattern, it stood out quickly.

That monitoring was not glamorous, and it was not new. It was operational knowledge turned into a guardrail. Without it, the polling loop could have continued until someone happened to notice an unusually large bill.

This is worth emphasizing because it is easy to see AI as the whole story. The model helped create the application. The monitoring, the historical baseline, and the people who responded to the alert are what limited the damage.

Working is not the same as economical

AI is good at getting to a plausible solution. It can write the query, connect the service, add a loop, and return the result. But it does not naturally care whether a process runs every minute instead of every hour, whether data can be cached, whether an existing system already solves part of the problem, or whether a warehouse is sized appropriately for the query.

Sometimes the generated solution will rebuild something that already exists. Sometimes it will choose a direct approach that is perfectly functional but wasteful at production scale. A polling loop is a simple example: it may make a feature feel responsive while quietly paying for repeated work that is unnecessary.

None of that makes the application useless, or AI a bad tool. A few years ago, a person without deep technical experience might not have been able to build and deploy this application at all. Now they can. That is real leverage.

But the leverage changes where the risk sits. Development time may go down while cloud spend, maintenance, security exposure, or reliability risk goes up. Those costs often arrive after the demo is working and the application is in use.

A short review can be a very good investment

The answer is not to ban AI-generated code or require every idea to go through a long development process. The answer is to put experienced eyes on the parts that determine how software behaves in the real world.

For a data-backed application, that review can be straightforward:

• What runs on a schedule, and how often does it actually need to run?
• Which queries execute, on what warehouse, and how much data do they scan?
• Can the application cache results, react to an event, or reuse an existing data set instead of polling?
• What is the expected cost at normal usage and at a failure mode?
• Which alerts will tell us when the application behaves differently from expected?

A review like this does not need to take longer than the work it is reviewing. In this case, it could have prevented a four-figure surprise. More importantly, it creates a habit of treating an AI-generated application as software that will be operated, not just code that needs to run once.

Keep the human in the loop

There is a familiar parallel with outsourcing. Lower-cost implementation can be a good trade when the work is understood and the output is reviewed. It becomes expensive when the apparent savings mean nobody owns the architecture, the quality, or the ongoing consequences.

AI assistance is similar. It can make capable builders out of more people, and that is something to embrace. But it can also produce slop, inefficiencies, security risks, and bugs that are easy to miss because the first version looks complete.

The goal should not be to slow people down. It should be to pair the speed of AI with monitoring that catches surprises and with people who understand the systems, costs, and tradeoffs behind the code.

This is the first in a series of examples from the gap between shipping software quickly and operating it well. AI can help you build faster. Make sure someone is also asking what the resulting system will cost to run.

Claude, Codex, and the rest are useful debugging partners. They can suggest hypotheses quickly, explain unfamiliar systems, and keep an investigation moving when you are stuck.

They can also send you on a very convincing rabbit hole.

We ran into that with a client’s ASP.NET application. After it had been up for a while, the first request for a static JavaScript file could be very slow. Requests after that were fast. It was the kind of narrow, intermittent behavior that invites a long list of theories.

Claude’s initial diagnosis was that SSL certificate revocation checking was holding up the first request. That is a real thing worth knowing about, and it sounded plausible in the abstract. But the application was using a self-signed certificate. There was no certificate authority revocation check to perform. A small piece of domain knowledge ruled out a direction that otherwise could have consumed hours.

The problem was not that Claude mentioned certificate revocation. The problem would have been treating a confident, technically detailed answer as evidence.

Start with what the system is doing

Rather than follow the SSL theory, we tested the behavior we could observe. We read the assets directly from the filesystem and fetched the asset through the application. The pattern was consistent:

1. Fetch an asset and the first request is slow.
2. Fetch it again immediately and it is fast.
3. Edit the file, then fetch it again, and the next request is slow again.

That is a much more useful description of the issue than “static JavaScript is slow.” The expensive path was associated with first access to changed file content. It was not ordinary request handling, and it did not fit the TLS explanation.

The experiment strongly pointed to endpoint scanning. SentinelOne was running in that environment and was the most likely cause: changed content was likely being scanned on its first access, while the next read benefited from the result already being available. We did not treat that as a definitive vendor-level attribution, but it fit the observed behavior far better than revocation checking did.

Plausible is not proven

LLMs are especially good at producing plausible explanations. They have seen the vocabulary around a symptom, and they can connect it to a real mechanism. That is useful for generating a list of things to investigate.

But a diagnosis has to survive the details of the actual system:

• Does the proposed mechanism exist in this deployment?
• Does it explain the timing and repeatability of the symptom?
• What inexpensive test could distinguish it from the other hypotheses?
• What observation would prove it wrong?

A self-signed certificate was enough to make us stop and question the revocation theory. The cold-read, warm-read, and modified-file test gave us a better hypothesis to pursue. Neither step required an encyclopedic knowledge of every possible cause. They required knowing enough to check the assumptions and to design a small experiment.

Use the model as a partner, not an authority

Claude still helped with the investigation. The right use was not to ask it for the answer and implement the first response. It was to use it as a partner while we compared theories against the environment and the measurements.

A practical debugging loop looks like this:

1. State the observation precisely, including what changes between a slow request and a fast one.
2. Ask the model for competing hypotheses and a test that would separate each one.
3. Check its assumptions against the architecture, configuration, and operational environment.
4. Run the smallest useful experiment.
5. Feed the result back in and repeat.

This is also why domain expertise still matters when using LLMs. If you cannot tell whether an answer fits the system you are operating, confidence and detail are easy to mistake for correctness. Bring in someone who knows the domain, or slow down enough to validate the model’s premises before chasing its conclusion.

The model can make a good investigator faster. It cannot replace the judgment needed to decide whether a theory belongs in the investigation at all.

Everybody likes an API. It is the clean version of an integration: documented endpoints, structured responses, credentials meant for software, and hopefully somebody else’s problem when the implementation changes.

But plenty of useful systems do not have one. Or they have an API that covers part of the product but not the screen a team actually needs. The data is there, behind a normal login, and somebody is opening a browser and copying it into a spreadsheet.

That is a good place for browser automation.

We have been looking at a practical combination of Playwright and Codex for this kind of work. Playwright drives a real Chrome browser. Codex can work with a browser exposed over the Chrome DevTools Protocol (CDP), which makes it useful for exploring an application and helping build the automation. The important piece is that the login stays human: someone who is authorized to use the site signs in and completes MFA themselves. The automated job uses the resulting browser session; it does not try to get around the login.

A concrete example is recording live odds from sportsbooks such as FanDuel or BetMGM. If an organization is allowed to collect and use the data, that is a much better job for a computer than for someone watching pages and updating a sheet all day.

The browser is sometimes the integration

This is not an argument to scrape everything. If there is an official API that does the job, use it. It is almost always less fragile and easier to support.

But sometimes the browser is the only interface available to the user. A person can log in, look at a live market, and see the numbers, but there is no supported endpoint for getting the same information into an internal system. In that situation, the browser can be the boundary between the site and your workflow.

There are obvious limits. The account needs to be authorized, and the intended use needs to comply with the site’s terms, contracts, and applicable law. That matters especially for sportsbooks, where access and permitted use can vary by operator and jurisdiction. This is not a way to bypass MFA, CAPTCHAs, rate limits, or other controls.

Getting a logged-in browser session

Playwright is a browser automation framework. It can launch Chromium or Chrome, navigate pages, click buttons, fill forms, and read what the page renders.

The feature that makes authenticated automation workable is a persistent browser profile. Instead of starting fresh every time, Playwright launches Chrome with the same profile directory. That lets it retain browser state such as cookies and sessions when the site permits it.

The first run is simple: launch a visible browser, log in normally, handle the two-factor prompt, and make sure the page you need is available. After that, the job can reopen that same profile. It can run visibly while somebody is building or debugging it, then run headlessly when it is ready to collect data on a schedule.

import { chromium } from "playwright";

const userDataDir = "/secure/path/to/browser-profile";
const context = await chromium.launchPersistentContext(userDataDir, {
  channel: "chrome",
  headless: false, // first run: let the account owner log in
});

const page = await context.newPage();
await page.goto("https://example.com/login");

// The authorized user completes sign-in and MFA in this browser window.

That profile directory is sensitive. It may contain an active session, so treat it like a credential: keep it in approved storage, restrict who can access it, and do not put it in source control or logs. Also plan for it to expire. The site decides how long a session lasts. If the job finds a login page again, it should stop and let a person reauthenticate.

Asking Codex to do the tedious part

The page is usually the part that makes these projects annoying. Modern web applications load data after the page appears, change the DOM as markets update, and use selectors that are not obvious until you can inspect the live application.

This is where Codex helps. With the authenticated browser connected over CDP, you can ask it to inspect the page, find the market and selection elements, and build the Playwright script around what is actually there. It is much faster than guessing at selectors from a screenshot or trying to reverse engineer an undocumented backend.

A reasonable first prompt is something like:

Set up Playwright with a persistent Chrome profile and launch it visibly so I can log in. Once I have an authenticated session, use that session to inspect the live odds page and build a script that records the event, market, selection, displayed odds, and collection time.

Codex is helping with the implementation; it is not replacing the account holder. Keep that line clear. It should not be asked to find credentials, solve MFA, or work around controls the site has put in place.

A live-odds collector

Say the job is to keep an internal, timestamped record of odds for a set of games and markets.

First decide what an observation is. At a minimum, it will probably include the sportsbook, event, market, selection, displayed odds, the source page, and the time the value was seen. That last field matters: a number collected at 2:00 PM is not the same thing as a number collected five minutes later.

Next, use the visible browser session to get to the right market and let Codex inspect the rendered page. The job needs to know when the market is actually loaded, how a suspended or unavailable price is represented, and which labels or attributes are stable enough to use as selectors. Prefer user-facing labels where possible over a long chain of generated CSS classes that will disappear in the next redesign.

Once that is understood, the collector can launch the same persistent profile in headless mode, visit only the pages it needs, validate what it finds, and write normalized records to a database or queue.

const context = await chromium.launchPersistentContext(userDataDir, {
  channel: "chrome",
  headless: true,
});
const page = await context.newPage();

await page.goto(targetMarketUrl, { waitUntil: "domcontentloaded" });
await page.getByRole("heading", { name: /live odds/i }).waitFor();

const collectedAt = new Date().toISOString();
const observations = await page.locator("[data-market]").evaluateAll((markets) =>
  markets.map((market) => ({
    market: market.getAttribute("data-market"),
    text: market.textContent?.trim(),
    collectedAt,
  }))
);

The code above is deliberately generic. The real selectors should come from the target site and should be tested against its actual states. The useful outcome is not just a script that reads a page once. It is a small internal data source that dashboards, reports, or models can rely on without each one having to understand the sportsbook’s UI.

The unglamorous stuff is what makes it work

A browser job will change when the website changes. That is normal. The difference between a useful integration and a fragile script is how it behaves on a bad day.

Keep the browser profile locked down. Save where each observation came from and when it was collected. Alert when the job suddenly gets no results, sees a login screen, or returns far fewer records than normal. And give somebody a straightforward way to rerun the visible browser and refresh the session.

That is enough to turn a manual task into something dependable without pretending the website is an API.

For companies sitting on useful data behind logins, this is a practical option: a person handles the authorization once, Playwright keeps the browser state, and Codex speeds up the work of turning what is on the screen into structured data.

There has been a lot of discussion recently about companies using AI to build internal tools that replace SaaS licenses. That is interesting, but it misses a big category of software: the web apps you cannot replace.

Sometimes the constraint is technical. More often, it is not. An insurance company may require you to use its verification portal. A specialty vendor may only accept orders through a clunky ecommerce site. Or a marketplace may be where all of the demand for your product or service lives.

You can build a better internal tool, but you still have to use those sites.

The browser is the integration point

Chrome extensions have always been a way to change the experience of a site you do not control. An extension can read information from a page, add controls to it, and help guide a user through a workflow.

Historically, that was possible but often not practical. You needed to write and maintain custom code for every awkward workflow, and the payoff had to be large enough to justify it.

The latest AI models change that calculation. It is now much easier to build an extension that augments a legacy site, whether that means changing a workflow, extracting information from a page, or adding LLM capabilities directly where people are already working.

Instead of asking someone to copy information from one system into another, you can put the assistance in the browser tab where the work already happens.

A bike search on Facebook Marketplace

I recently had a good excuse to try this out. I was looking for a bike on Facebook Marketplace with a specific set of requirements. The hard part was not finding listings. It was reviewing the photos for each listing to determine whether a bike was actually a fit.

Doing that manually meant opening and reviewing dozens of listings every day. That is exactly the sort of repetitive visual task that an AI model can help with.

So I built a Chrome extension that uses OpenAI to review listing photos and flag the listings that match what I was looking for. Rather than replacing Facebook Marketplace, the extension improves the part of the Marketplace workflow that was taking the most time.

The result is not a fully autonomous bike buyer. It is a faster way to narrow down a large list of listings so I can spend my time looking at the promising ones.

Where this approach works

The Marketplace example is personal, but the pattern applies to business workflows too. Look for web-based processes where a person repeatedly has to review, classify, summarize, or move information before they can make a decision.

A Chrome extension can be a practical place to add help to:

• an insurer’s required portal
• a vendor ordering site
• a marketplace your team depends on
• an internal legacy application that is difficult to change

The goal is not necessarily to replace the site. It is to remove the tedious steps around it while keeping people in the workflow they already need to use.

See it in action

Check out the demo below to see the bike finder at work:

Watch the video on YouTube

Interested in building something similar for a workflow your team cannot avoid? Get in touch with Setfive.

Inside most companies right now, AI is already at work. Even if you have not rolled out ChatGPT Team or Copilot, people are using their personal subscriptions to speed things up. That is great for initiative, not so great for consistency. It looks a lot like the early Excel era: clever workarounds, duplicate effort, and new questions about data governance.

Sherpa is our way to bring order to that energy. We analyze real tasks from Asana or monday.com, group similar work with an LLM, and point to the places where automation will pay off. You get a clear plan you can act on, without buying another stack of licenses first.

What Sherpa is

Think of Sherpa as an AI audit for your task data. It connects to your workspace, reads tasks with your permission, and maps the repetitive patterns that eat time. Then it scores where automation is likely to win, explains why, and recommends how to build it. The output is practical and specific: plays, tools, prompts, and an effort estimate so you can prioritize.

How it works

You start by connecting Asana or monday.com with OAuth. Access is read only and under your control at all times. We do not change or write tasks.

Next, a large language model groups related work and finds recurring patterns. That includes obvious repeats, quiet duplicates that happen across teams, and tasks that often move together in a process.

Finally, we deliver a short report that tells you what to automate, in what order, and how. Each recommendation includes expected time savings, suggested connectors or integrations, and sample prompts so your team can move quickly.

Typical turnaround is about a week from connection to findings.

Why scan tasks now

Personal AI usage is already shaping how work gets done. Sherpa helps you see what is working, what is risky, and what should be standardized. It replaces guesswork with a picture of real workflows, so you can invest in the right automations and avoid paying for licenses that will not get used.

Leaders also get a common view of where hours are going. That makes process conversations easier. Instead of debating tools in the abstract, you can point to specific clusters of tasks and decide how to fix them.

What you get in the report

• An automation scorecard with high, medium, and low opportunities, each with a short rationale.
• A top 3 list of automation plays with exact steps, recommended tools, and integration notes.
• An impact section that translates hours into dollars using your inputs.

You also get a recurring task map, duplicate detection across teams, suggested prompts and connectors, and a next step build plan that you can implement with your team or with Setfive.

A sample finding

Manual reporting shows up in almost every audit. A team exports CSVs every Friday, merges them by hand, and posts a summary. The play is straightforward: schedule the extract, load it to a source of truth, and send a templated summary to Slack or email.

• Impact: High
• Effort: Medium
• Estimated savings: 6 hours per week

If 10 people each save 6 hours per week at an average loaded rate of 75 dollars per hour, that is 6×10×75=4500 dollars of capacity back every week.

Where Sherpa fits with ChatGPT Team and Copilot

Already have licenses? Sherpa shows where to deploy them and turns ad hoc prompts into repeatable, auditable workflows.

Still evaluating? Run Sherpa first to find the highest value use cases, then buy only what you need.

Not ready to buy seats? Many plays use tools you already have, so you can capture savings now and expand later.

Security and privacy

Sherpa uses OAuth with scoped, read only access. You can revoke access at any time. We follow your data retention requirements, and your findings are your IP. We do not use your data to train public models.

Who benefits

Ops and RevOps leaders with checklist heavy processes. PMOs juggling handoffs. CS and Support teams producing weekly reports. Marketing ops moving content through approvals. Finance and People teams closing the loop on routine reconciliations. If the same task shows up again and again, Sherpa will find it.

FAQs

Do we need to change how we work to try it? No. Sherpa analyzes the work you already do.

Will this replace people? The goal is to remove low leverage, repetitive tasks so your team can focus on higher value work.

Can you help implement the plays? Yes. Implementation projects are scoped after the audit.

Try the Free AI Task Audit

Stop guessing where AI will help. Measure it. Sherpa shows you the work your team should not be doing and how to automate it, fast.

Get your Free AI Task Audit, a concise scorecard, and a prioritized plan with savings you can defend.
Ready to see your opportunities? Get in touch at contact@setfive.com or read more about Sherpa at sherpa.setfive.com