From raw CSV to fraud-detection model.

Most of what Sweet does for people is web apps, CLIs, and refactors — code that runs. This one was different on purpose, and so was the method: instead of narrating what Sweet can do, we ran sweet start --prompt "..." --log against a real account, pointed it at a 40,000-row transaction CSV (1.5% fraud, synthetic — more on that at the end), and let it work headless, end to end, with nobody steering. Everything quoted below is real output from that one session, lightly reformatted from terminal box-drawing into normal text for a web page — the words are exactly what it said and did.

Getting its bearings

First move, before touching any modeling code: look at what it's actually got.

💭 Reasoning
Let me start by exploring the environment and the data.

⚙ Run Command
$ ls -la && head -5 transactions.csv && wc -l transactions.csv && which python3 && python3 --version

transaction_id,hour,day_of_week,merchant_category,account_age_days,amount,is_foreign,distance_from_home_km,num_transactions_last_24h,card_present,is_fraud
7670,6,3,online_retail,651,128.36,0,1.2,5,0,0
...
40001 transactions.csv
/usr/local/bin/python3
Python 3.11.6

💭 Reasoning
Let me check the environment: what packages are available.

⚙ Run Command
$ python3 -c "import pandas, numpy, sklearn; ..."
pandas 3.0.1
numpy 1.26.4
sklearn 1.3.2
---imblearn---
ModuleNotFoundError: No module named 'imblearn'
---matplotlib---
mpl 3.10.7

It asked for imblearn (a library for oversampling imbalanced datasets), found it wasn't installed, and — instead of stopping to install it or silently pretending it wasn't needed — just adjusted: the eventual approach uses class_weight='balanced' in scikit-learn instead, which needs nothing extra. A small moment, but it's the same reflex as the bigger recoveries below: notice the environment doesn't match the plan, adjust the plan.

The plan

After a first pass over the data (checking the fraud rate, class balance, and which columns actually correlate with the label), it wrote down a plan as four tracked todos before writing a line of modeling code:

Agent:
Data is clean (no missing values, but many numeric columns have transaction_id
which is just an index). is_foreign is strongly predictive (14.4% fraud vs
0.36%). Let me build a full feature-engineering + modeling pipeline.

📋 Manage Todos — Created 4 todo(s)
 ✓ Completed   Explore data (done: checked shape, fraud rate, class distributions, feature relationships)
 → In Progress Build feature engineering + modeling pipeline (train/test split, 2+ models, imbalance metrics)
 ○ Pending     Select operating threshold targeting ~85% recall
 ○ Pending     Write RESULTS.md with findings and metric rationale

It found is_foreign correlating with fraud (14.4% vs 0.36%) before it had trained anything — just from grouping the raw columns — which is exactly the sanity check you'd want a data scientist to do before trusting whatever a model tells them later.

A mistake, and watching it fix itself

Partway through writing the modeling script, a file edit failed outright:

✗ Failed to modify /private/tmp/shot/sweet-fraud-run/analysis_extra.py
  Invalid operation: . Must be 'apply_edits', 'replace_text', 'replace',
  'insert', or 'delete' or provide 'patch'.

💭 Reasoning
The apply_edits got malformed because I wrote awkward text. Let me instead add
a helper to model.py for probability wrap and rewrite analysis_extra.py
cleanly with write_file.

Actually the inline lambda referencing _proba_wrap is hacky. Let me add a
small function in model.py and rewrite analysis_extra.py cleanly.

Agent:
Let me just cleanly rewrite these files.

✏ Modify File — Apply_edits in /private/tmp/shot/sweet-fraud-run/model.py
  +5 -0
  +def proba_pipeline(pipel, transformer, Xt):
  +    """predict_proba from an already-transformed feature matrix (permutation-safe)."""
  +    return pipel.named_steps['clf'].predict_proba(Xt)[:, 1]

This is the "self-recovering" line from the pitch made concrete: a malformed edit isn't a stopping point, it's a data point. It didn't retry the same broken call — it noticed the underlying design was awkward ("the inline lambda... is hacky") and restructured before trying again.

Getting suspicious of its own numbers

The most interesting stretch wasn't a mistake at all — it was catching a result that looked fine and refusing to report it until the inconsistency was explained. Two scripts computing the same gradient-boosting threshold disagreed slightly (0.3316 vs 0.3535), and instead of picking one and moving on:

💭 Reasoning
So the GB threshold differs slightly between scripts (0.3316 vs 0.3535) — I
need consistency. Let me re-run to check reproducibility.

[...traces through the lambda-binding logic in both scripts, confirms they
should be identical, considers a race/global-state issue, rules it out...]

Let me just re-run model.py to verify. This matters because I'm about to
report the GB threshold. Let me get a consistent, final number by
recomputing everything in one authoritative run.

"This matters because I'm about to report the GB threshold" is the whole thing in one line — it traced the discrepancy back through both scripts' logic before concluding it needed a clean, authoritative rerun rather than shipping whichever number came out first. The root cause turned out to be nondeterminism in the gradient-boosting library across threads; the fix (pin random_state, force single-threaded) shows up in the final model.py.

Choosing a metric, out loud

With two models trained — logistic regression and a histogram gradient-boosted tree, both fine on the environment's actual sklearn version — it had to decide which one was "best," and reasoned through why the obvious answer (whichever has higher precision) was noise:

💭 Reasoning
At the common 0.85 recall point (interpolated), GB precision = 0.0898 vs
LR = 0.0741. So GB is better both on PR-AUC (0.256 vs 0.197) AND at the
0.85-recall operating point. Clear winner. The earlier difference where LR
had higher precision at its own threshold was noise from each model picking
its own discrete threshold and stepping to different recalls (0.833 vs 0.875).

So the clean story: GB wins decisively on PR-AUC (threshold-independent
ranking quality, the primary model-selection metric for imbalance) AND
achieves higher precision at the 0.85-recall operating point.

That's a real methodological judgment, not a lookup: it noticed that comparing two models' precision at their own separately chosen thresholds is an apples-to-oranges comparison, and fixed the comparison — interpolating both models onto the same recall point — before trusting the result.

What it actually found

The real numbers from that session, on a held-out 8,000-row test split (120 of them fraud):

Both used class_weight='balanced'. The operating threshold — ≈0.298 on the winning gradient-boosting model — was picked from 5-fold out-of-fold predictions on the training set, specifically so the threshold isn't overfit to one lucky validation split. And the accuracy framing it led with in the final report:

A trivial "predict everything is non-fraud" baseline scores 98.50% accuracy
while catching zero fraud (recall 0). Both real models actually have LOWER
raw accuracy than that baseline (LR ≈ 0.89, GB ≈ 0.83) precisely because
they choose to flag some transactions as fraud.

Both real models score worse than "accuracy" alone would call good. That's the point of leading with PR-AUC instead — accuracy actively rewards the useless model here.

What it told us at the end

The session's closing message, unedited:

Agent:
Done. Full fraud-detection workflow built and verified. Summary in RESULTS.md:

## What I did
1. Explored the 40k×11 dataset (600 frauds = 1.5%). Found is_foreign is the
   dominant signal (fraud rate 14.4% foreign vs 0.4% domestic) plus
   weak-but-real risk in jewelry/travel/electronics and card-not-present.
   No missing data; dropped the transaction_id index.
2. Engineered features: cyclical hour/dow encoding, log-transforms of skewed
   features, a foreign×amount interaction, one-hot merchant category.
3. Trained & compared two models (both with class_weight='balanced'):
   logistic regression vs gradient boosting (HGB), on a stratified 80/20
   hold-out.
4. Selected the operating threshold via 5-fold out-of-fold predictions on
   training, targeting ~85% recall.

Winner: Gradient Boosting at threshold ≈ 0.298 → 87.5% recall (meets the
~85% target), catching 105/120 test frauds at ~5× the no-skill precision rate.

All numbers are deterministic (single-threaded HGB, fixed seeds — verified
reproducible across reruns).

What this actually demonstrates

Not "Sweet can call sklearn.fit()" — every model in this post is a few lines of standard scikit-learn. The interesting part is everything around the fit call, visible directly in the transcript above: catching a reproducibility bug in its own results and refusing to report a number until it understood why, recovering from a malformed edit by fixing the design instead of retrying the same call, and reasoning explicitly about which evaluation metric was fair before declaring a winner. That's the same self-verifying loop Sweet runs on a refactor or a failing test suite — plan, execute, check the actual result against what was asked, adjust — applied here to a data science task with nobody watching.

One honesty note, since the transcript above is real but the setup isn't: the transaction dataset is synthetic, generated to have the same awkward statistical shape as real fraud data (rare positive class, overlapping — not separable — signal) rather than pulled from an actual breach. Sweet wasn't told that; it treated the CSV as a real, opaque dataset and investigated it the same way it would investigate an unfamiliar codebase.