Skip to content

About

A Multimodal AI system built with Python and Streamlit that provides a single interface for text generation, image creation, image captioning, speech-to-text, text-to-speech, and video description using open-source models like FLAN-T5, Stable Diffusion, BLIP, Whisper, OpenCV, and gTTS—without any paid APIs or external keys.

Resources

Stars

1 star

Watchers

0 watching

Forks

Latest commit

 

History

1 Commit

Folders and files

NameName
Last commit message
Last commit date
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

Multimodal AI System

Overview

This project is a multimodal AI application developed using Python and Streamlit. It supports multiple input and output modalities, allowing users to interact with AI through text, images, audio, and video.

The system uses open-source models and libraries without requiring any paid APIs or API keys.


Features

  • Text → Text Generation
  • Text → Image Generation
  • Image → Text Captioning
  • Text → Audio Conversion
  • Audio → Text Transcription
  • Video → Text Description

Models Used

Modality Model/Library
Text → Text google/flan-t5-base
Text → Image runwayml/stable-diffusion-v1-5
Image → Text Salesforce/blip-image-captioning-base
Text → Audio gTTS
Audio → Text OpenAI Whisper
Video → Text BLIP + OpenCV

Technologies

  • Python
  • Streamlit
  • Hugging Face Transformers
  • Diffusers
  • PyTorch
  • OpenCV
  • Whisper
  • gTTS

Project Structure

assignment_3/
│
├── assets/
├── models/
│   ├── text_model.py
│   ├── image_model.py
│   ├── caption_model.py
│   ├── tts_model.py
│   ├── whisper_model.py
│   └── video_model.py
│
├── notebook/
├── outputs/
├── app.py
├── requirements.txt
└── README.md

Installation

  1. Create a virtual environment:
python -m venv venv
  1. Activate the environment:
venv\Scripts\activate
  1. Install dependencies:
pip install -r requirements.txt

Running the Application

streamlit run app.py

Supported Inputs

  • Text Prompts
  • Image Uploads (JPG, JPEG, PNG)
  • Audio Uploads (MP3, WAV)
  • Video Uploads (MP4, AVI, MOV)

Applications

  • AI Chat Assistant
  • Image Captioning
  • Speech Transcription
  • Text-to-Speech Conversion
  • Video Content Understanding

Conclusion

This project demonstrates the implementation of a multimodal AI system capable of processing and generating information across text, image, audio, and video modalities using open-source AI models and a Streamlit-based web interface.


Author

** Nimish Pathak **

About

A Multimodal AI system built with Python and Streamlit that provides a single interface for text generation, image creation, image captioning, speech-to-text, text-to-speech, and video description using open-source models like FLAN-T5, Stable Diffusion, BLIP, Whisper, OpenCV, and gTTS—without any paid APIs or external keys.

Resources

Stars

1 star

Watchers

0 watching

Forks

Releases

Packages

Contributors

Languages