We’ve received a lot of questions about how we built this, so we decided to share it with you. The use case is quite strong and goes beyond just recipe extraction. If you find this article helpful, feel free to share it with your fellow developers!

What you need first

What makes our extraction work comes down to two main components:

1. Scraper: First, we need to get the data. By data, we mean raw content—whether it’s website content (e.g., HTML) or videos (from YouTube, TikTok, Instagram, etc.). You can either build your own scraper or use third-party services like Apify.

2. LLM Model: Once you have the data, you’ll need an LLM to extract the recipe content, remove unnecessary parts, and output everything in a structured format (e.g., JSON).

So first, decide whether you want to build these components yourself or use existing third-party providers. Once you have both in place, you’re ready to move to the next step.

Building the extraction module

Level 1: Generic Website

1. When a user inputs a URL from the client, it invokes a Supabase serverless function. This function sends the request to our server (FastAPI), which then fetches the data using our scraper (e.g., Firecrawl).

2. Once the scraper returns the result, we send the HTML content to our LLM model (e.g., Gemini).

3. After processing, the LLM returns the structured output. We parse it on our server and finally send it back to the client.

Some caveats:

1. Some websites block web scrapers.

Level 2: Video (with audio)

1. This is more or less the same as Level 1, except we have additional steps.

2. When a user submits a URL, it goes through Supabase to our server. To fetch the video content (whether from TikTok or another platform), we use Apify as our scraper.

3. Apify usually returns a lot of data, but what matters most to us are the audio_url or video_url, and the post description.

4. On our server, we first download the audio from the audio_url (or extract audio from the video_url). We then send the audio to our LLM model for transcription.

5. Once we obtain the transcription, we combine it with the post description and send both to the LLM again—this time to extract the recipe.

6. After receiving the structured output (in JSON format), we process it on our server and send it back to the client.

Some caveat(s):

1. This only works for videos that contain verbal explanations of the steps and ingredients. If the video only has background music and no spoken content, nothing can be extracted.

2. Why don’t we just send the entire video to the LLM? Because transcribing video directly consumes a LOT of tokens—and honestly, that gets expensive. That’s why we extract and send only the audio instead.

Level 3: Video (without audio?)

At the moment, this is still in development. I'm trying to find the most efficient and cost-effective way to tackle this problem. Will update once we found a solution for this =)

Legal Considerations

1. This process can be sensitive since you are using someone else’s content. If you plan to sell or redistribute the extracted content, you may run into legal issues without the owner’s permission.

Closing

Now you should have a clearer idea of how to build your own video content extractor. Don’t limit this approach to recipe extraction—I can see many other potential use cases for this module.

Hope this article helps. Have fun building! 🚀