This project is a multimodal AI application developed using Python and Streamlit. It supports multiple input and output modalities, allowing users to interact with AI through text, images, audio, and video.
The system uses open-source models and libraries without requiring any paid APIs or API keys.
- Text → Text Generation
- Text → Image Generation
- Image → Text Captioning
- Text → Audio Conversion
- Audio → Text Transcription
- Video → Text Description
| Modality | Model/Library |
|---|---|
| Text → Text | google/flan-t5-base |
| Text → Image | runwayml/stable-diffusion-v1-5 |
| Image → Text | Salesforce/blip-image-captioning-base |
| Text → Audio | gTTS |
| Audio → Text | OpenAI Whisper |
| Video → Text | BLIP + OpenCV |
- Python
- Streamlit
- Hugging Face Transformers
- Diffusers
- PyTorch
- OpenCV
- Whisper
- gTTS
assignment_3/
│
├── assets/
├── models/
│ ├── text_model.py
│ ├── image_model.py
│ ├── caption_model.py
│ ├── tts_model.py
│ ├── whisper_model.py
│ └── video_model.py
│
├── notebook/
├── outputs/
├── app.py
├── requirements.txt
└── README.md
- Create a virtual environment:
python -m venv venv- Activate the environment:
venv\Scripts\activate- Install dependencies:
pip install -r requirements.txtstreamlit run app.py- Text Prompts
- Image Uploads (JPG, JPEG, PNG)
- Audio Uploads (MP3, WAV)
- Video Uploads (MP4, AVI, MOV)
- AI Chat Assistant
- Image Captioning
- Speech Transcription
- Text-to-Speech Conversion
- Video Content Understanding
This project demonstrates the implementation of a multimodal AI system capable of processing and generating information across text, image, audio, and video modalities using open-source AI models and a Streamlit-based web interface.
** Nimish Pathak **