A Skincare Chatbot, 120,000 Reddit Comments, and the Classification Problem That Ate a Project

In November 2024, a developer fine-tuned a small language model on r/SkincareAddiction. The first version answered like a redditor: ask it for a routine and it told you to read the sidebar. Sometimes it insulted you. The model was only as good as its training data, and the training data was 120,633 Reddit comments that needed to be filtered down to the useful ones.

The project became a case study in how data classification can consume more time than the thing you are actually building. It also became a comparison point for a newer tool called Jev, a general decision model that handled the same classification work in 23 minutes for $3.47, replacing weeks of manual pipeline construction.

Building a Teacher-Student Classifier Pipeline

Fine-tuning an instruct model needs input-output pairs. The developer used Reddit post titles and bodies as inputs and comments with at least two upvotes as outputs. Whatever a well-liked Reddit comment does, the model learns to do. The problem was that not all upvoted comments are good training data.

Three questions needed answers for every comment: Is it useful, meaning does it contribute to skincare discussion? Is it objective, meaning is it a personal anecdote rather than general advice? Is it quality, meaning is it spam, is it answering the question, and is it respectful?

The developer tried several approaches. Regex patterns were too leaky for semantic differences. Vectorizing comments and clustering them produced results that were hard to interpret. Sending every comment to an LLM directly would have cost between $70 and $180 at November 2024 prices. The chosen method was a teacher-student pipeline: a large, expensive LLM labels a sample, a tiny transformer learns those labels, and the small model processes the full dataset.

Each step was its own sub-project. The developer manually graded a few hundred comments, iterated on prompts with Claude until the labels matched, trained the classifier, fine-tuned the LLM, tested it, and repeated. Binary choices with confidence ratings turned out to be easier to train and set cutoffs on than the original 1-to-5 scoring system. Different teacher and classifier models were used for different steps, mostly for cost and performance reasons.

Acquiring teacher labels cost about $7 per pass. The classifier models agreed with the teacher up to 97% of the time on usefulness. But the teacher itself was not producing perfect labels, creating a ceiling on the whole approach.

Jev Removes the Pipeline

Jev is a general decision model, not an LLM. It takes data and questions and returns labels with confidence scores, no labeling, no fine-tuning, no training pipeline. The developer sent all 120,633 comment pairs through Jev with the three classification questions, five records per request. Larger batches timed out or produced worse answers. The five-record batch size agreed with single-record calls about 97% of the time.

The full pass took 23 minutes and cost $3.47. Against the old teacher-labeled dataset, Jev agreed 86% of the time on usefulness, 73% on objectivity, and 88% on quality. The handmade classifiers scored higher on raw accuracy, but the teacher labels they were measured against were imperfect too. An independent comparison using Gemini 3.8 Flash found Jev was better on objectivity and similar on the other categories.

The real advantage was not accuracy but iteration speed. With Jev, the developer could change the classification prompt and rerun the full dataset in minutes. The old pipeline required regrading samples, retraining classifiers, and retesting, each step a separate project.

What the Cost Comparison Shows

At the developer's scale, the cost difference between approaches was not dramatic. Jev cost $3.47 for the full dataset. Gemini 3.5 Flash-Lite would have cost about $22 projected. Gemini 3.8 Flash about $46. Claude 3.5 at November 2024 prices would have been $70 to $180. The old teacher-student approach cost about $7 for labels plus $1 to $2 in GPU compute per classifier per round.

The cost mattered less than the time. The developer estimated that weeks could have been spent perfecting the classifier alone. Each optimization, testing different teacher models, adjusting confidence thresholds, choosing between single and multi-classifier architectures, was its own project. Jev collapsed that into a single API call with a prompt.

The developer noted that the distraction of data filtering meant other problems went unaddressed, like handling out-of-distribution data around luxury products or non-Western skincare brands. The classification pipeline was the side quest that consumed the project. For future work, the developer plans to use Jev to skip that step entirely and focus on the actual product.