How I Increased Speed by 3X and Reduced Costs by 90X

I recently hit a wall with my project, Little Stories.

The app is simple: a child provides a theme, and the AI generates a 10-page rhyming book with custom illustrations. It worked beautifully, but there was a hidden "AI tax" lurking in my dashboard. Every time a single book was generated using a standard GPT/Gemini pipeline, it cost about $1.35.

If you're building a consumer app, that isn't just "expensive"—it's a death sentence for your unit economics.

I decided to run an experiment: could I maintain the quality while reducing the cost to the absolute bottom line? The result was a refactor that dropped the price per book from $1.35 to roughly $0.015.

Here is how I engineered a 90X cost saving while making the user experience feel 3X faster.

The Problem: The "Wait-for-All" Monolith

Most AI apps follow a standard synchronous request-response pattern. You hit an endpoint, the server calls an LLM, then it calls an image generator ten times, and finally, it sends a massive JSON back to the client.

In Little Stories, this meant:

1. Text Gen: 5 seconds.
2. 10 Images (Sequential): ~60-90 seconds.
3. Network Overhead: ~5 seconds.

The user sat staring at a loading spinner for nearly two minutes. In the world of modern UX, two minutes is an eternity.

The Experiment: Finding the "Bottom Line"

To fix the economics, I stepped outside the premium ecosystem and looked for "price-leader" open-source providers. I landed on a specific combination of models that specialize in speed and efficiency:

• The Poem (Text): I swapped the "Pro" models for Llama 3.1 8B (via Groq). At $0.05 per 1M tokens, the text for a whole book now costs $0.0001.
• The Scenes (Images): I moved to SDXL Lightning (via Fal.ai). This is a distilled model that generates high-quality images in just 2-4 steps. At $0.001 per image, the artwork for a 10-page book now costs $0.01.

Total "Bottom Line" Cost: ~$0.015 per book.

The Solution: Progressive Loading with SSE

Saving money is one thing; improving the experience is another. To solve the 90-second wait, I refactored the backend to use Server-Sent Events (SSE) and Async Generators.

Instead of one big package, the server now "streams" the book page-by-page.

1. The Priority Burst

As soon as Groq finishes the text (which takes less than 2 seconds), the server sends a text:ready event. The frontend immediately renders the book layout. The user can start reading the title and first stanza within seconds.

2. Parallel "Chunks"

While the user is reading Page 1, the server fires off image requests in parallel batches of three. Because SDXL Lightning is so fast (~0.18s per image), the first illustration pops into place before the reader even finishes the first stanza.

Final Thoughts

We often default to the "big" models because they are easy to implement. But as developers, our job is to find the intersection of high quality and sustainable cost.

By moving to a streaming architecture and leveraging distilled open-source models, I didn't just save $1.33 per book. I built a more responsive, magical experience that allows the story to "illustrate itself" right before the child's eyes.

The lesson? Don't wait for the final response. Stream the journey.