Image Caption Generator
VisionCaption AI is a deep-learning application that combines InceptionV3 (CNN) for image feature extraction and an LSTM (RNN) decoder within an overall ANN architecture to instantly generate accurate descriptions for any uploaded photo.
Tech stack
Impact
Problem Statement
Creating meaningful descriptions for images is a complex task requiring both computer vision to understand the image content and sequence modeling to articulate it. The goal was to build an accessible web app utilizing ANN, CNN, and RNN architectures to bridge this gap.
Dataset
Trained on the Flickr8k dataset, comprising 8,000 images each paired with five different human-annotated captions to teach the model varied descriptive language.
Architecture
A hybrid neural network architecture where a pre-trained CNN (InceptionV3) extracts a 2048-dimensional feature vector from the image, which is then passed into an RNN (LSTM network) that generates the caption word by word.
Model Selection
CNN (InceptionV3) was chosen for its excellent feature extraction capabilities on images, and RNN (LSTM) for its ability to remember sequence context, making the combined ANN ideal for text generation from images.
Training Process
The model was trained using categorical crossentropy loss to predict the next word in the sequence. A custom tokenizer with an 8.8K vocabulary was built to map words to integer sequences.
Evaluation Metrics
Results
A live Streamlit application (VisionCaption AI) where users can upload any JPG or PNG image and receive an instantly generated, deep-learning powered description of the scene.
Key Learnings
- 1Combining CNNs and RNNs (LSTMs) is highly effective for multimodal tasks like image captioning.
- 2Pre-trained models like InceptionV3 significantly reduce training time and improve feature quality.
- 3Streamlit provides an excellent, rapid way to deploy complex deep learning models into interactive web apps.
Want to dig deeper?
Explore the code, or get in touch if you'd like to talk through the approach.