Why I Built Street Talk English

A few weeks ago, I came across the #AllThingsAgentic Hackathon. I had been thinking about the gap between learning English and actually using it in daily life, so I discussed the idea with my wife and decided to give it a try.

That decision became Street Talk English: an agentic English-practice experience that turns a situation a learner genuinely needs to handle into a personalised speaking mission.

Watch the Street Talk English demo on YouTube →

The problem I wanted to solve

Most English learners do not struggle because they have never seen a grammar rule. They struggle because a real conversation arrives before they feel ready: ordering at a café, asking for help at a service counter, or discussing an important decision at home.

Traditional exercises can be useful, but they often begin with an imaginary situation. I wanted to build something that begins with a situation the learner actually cares about, gives them a safe space to practise, and helps them remember the useful language when the real moment arrives.

The goal is not to replace human conversation. It is to help learners feel prepared enough to have it.

From a real situation to a speaking mission

In Street Talk English, a learner can start with a ready-to-practise coffee-shop roleplay. They can also upload a non-sensitive real-world image and add a short context note.

For example, I created a mission around a kitten at an adoption event. The learner’s goal is to discuss adopting that kitten with Alice. The situation feels more personal than a generic “convince someone” exercise because it has real stakes: daily responsibility, a potentially messy home, and vet bills.

Street Talk creates a structured mission with:

• A practical communication goal
• A conversation partner and their perspective
• Useful phrases to draw on
• Clear, measurable objectives

I made the learner review the mission before starting. That matters to me because I wanted the AI to feel like a collaborative preparation partner, not an unpredictable conversation generator.

Practising the conversation

Learners can reply by voice or text. During the roleplay, the partner stays in character, responds naturally to imperfect English, and asks relevant follow-up questions.

In the kitten scenario, Alice may ask who will feed and clean up after the kitten, how the learner will contribute to costs, and what compromise might make the decision feel reasonable. To complete the mission, the learner needs to make a real case: explain why this kitten needs a home, describe a care schedule around work, offer a financial plan, and negotiate a trial period or an extra chore.

I did not want the learner to chase perfect grammar. I wanted them to practise communicating a complete, believable plan.

Making progress visible

As the conversation progresses, Street Talk tracks practical objectives. After the mission, the results view shows what the learner communicated successfully and what still needs more practice.

That progress is stored and can shape future missions. My intention was to make practice cumulative: instead of starting from zero each time, learners can build confidence through a history of real communication goals they have already worked through.

How I built it on Google Cloud

I deployed Street Talk English as a containerised application on Google Cloud Run. The React front end and FastAPI backend work together as one protected learner experience.

I use Google ADK to orchestrate the roleplay workflow. Gemini 3.5 Flash on Vertex AI creates personalised missions from the learner’s image and context, supports the in-character dialogue, and evaluates objective evidence. Firestore stores persistent learner progress and mission state so the experience can adapt over time.

I also added Google Cloud Speech-to-Text and Text-to-Speech for voice practice. I designed the app with privacy boundaries in mind: raw images and raw audio are not retained or logged as learner history.

Why I added video and music

After a learner completes an eligible image-based mission, Street Talk can offer two optional completion experiences. Both require explicit learner consent and are separate from the core learning flow.

First, Veo 3.1 Fast can create a short private audiovisual recap. I see this as a way for learners to revisit the setting, the communication goal, and the confidence they built during the roleplay.

Second, Lyria 3 Clip can create a short private memory song. Gemini first creates concise lyrics grounded in the learner’s verified mission evidence, then Lyria turns those lyrics into music. I am not presenting this as a guarantee that music improves retention. I see it as a memorable recall aid.

For the kitten mission, the song becomes a light-hearted verbal contract. Its chorus can remind the learner to mention daily care, financial contribution, and the compromise they agreed to propose. When it is time to speak with Alice in real life, they have a compact mental structure to lean on.

What I learned from building it

This project helped me think more carefully about what an AI learning partner should do. The most important part is not generating a clever answer. It is helping a learner prepare for a conversation that matters to them, while keeping the experience focused, measurable, and respectful of their privacy.

Street Talk English is my attempt to make English practice feel closer to real life: turn a real situation into guided practice, show progress, and help learners remember what to say when the moment matters.

Turn real life into confident English.