02-image-caption-generation-vision-encoder-decoder

Image Caption Generation with Vision Encoder-Decoder

An end-to-end computer-vision and natural-language-generation project that combines a Vision Transformer encoder with a GPT-2 decoder, evaluates pretrained and project-fine-tuned captioning behavior, exports the model to browser-compatible ONNX, validates PyTorch-to-ONNX parity, and publishes a free client-side image-captioning application through Hugging Face Static Spaces.

Status: Portfolio-ready, evaluated, model artifacts published, and live application deployed
Live application: Open the Image Captioning Static Space
Published model: Open the Vision Encoder-Decoder ONNX model repository
Primary stack: Python · PyTorch · ViT · GPT-2 · Hugging Face Transformers · ONNX · ONNX Runtime Web · Transformers.js · JavaScript · Hugging Face Spaces

---

Responsible Use

This project is intended for education, technical learning, experimentation, and portfolio demonstration.

• Generated captions may be incomplete, inaccurate, biased, overly generic, or unrelated to important visual details.
• A caption is a model-generated interpretation, not verified ground truth.
• The system may miss small objects, text inside images, spatial relationships, unusual scenes, and culturally specific context.
• Do not use generated captions as the sole basis for accessibility, medical, legal, safety-critical, surveillance, identity, compliance, or business-critical decisions.
• Do not upload private, confidential, sensitive, regulated, proprietary, or personally identifiable images to the public application.
• The application does not identify people and should not be used for identity recognition.
• Human review is required before using a caption in any consequential workflow.

---

Business Problem

Images are common in product catalogs, inspection records, media collections, quality documentation, e-commerce systems, accessibility workflows, search engines, content-management platforms, and analytics pipelines. Manually writing descriptions for large image collections is repetitive, expensive, and difficult to scale.

This project investigates:

Can a Vision Encoder-Decoder model transform an input image into a useful natural-language caption while exposing decoding behavior, evaluation results, deployment trade-offs, and browser-compatible inference?

The system supports:

• Sample-image selection
• Custom JPG and PNG upload
• Pretrained and project-fine-tuned model options
• Greedy decoding
• Beam-search decoding
• Caption comparison
• Token-generation traces
• Attention-oriented visual inspection
• Evaluation summaries
• Browser-side ONNX inference
• Model and deployment disclosure

---

Project Objective

Build a professional image-captioning solution that can:

1. Load and validate image-caption data for model development.
2. Preprocess RGB images for a Vision Transformer encoder.
3. Tokenize reference captions for an autoregressive GPT-2 decoder.
4. Fine-tune a Vision Encoder-Decoder model using GPU acceleration.
5. Compare the pretrained checkpoint with the project-fine-tuned checkpoint.
6. Evaluate caption quality using BLEU, METEOR, ROUGE-L, CIDEr, and CLIPScore.
7. Measure caption diversity, repetition, latency, and model size.
8. Support greedy and beam-search generation.
9. Export browser-compatible encoder and decoder graphs to ONNX.
10. Quantize ONNX artifacts for client-side deployment.
11. Validate PyTorch-to-ONNX output parity.
12. Publish the deployment artifacts through Hugging Face Model Hub.
13. Deploy a free static browser application through Hugging Face Spaces.
14. Preserve honest model limitations and performance trade-offs.
15. Demonstrate applied Computer Vision, NLP, multimodal AI, evaluation, and deployment skills.

---

Project Pattern

Item	Implementation
Project number	02
Project name	02-image-caption-generation-vision-encoder-decoder
Application	Select or upload an image and generate a caption
Core architecture	Vision Transformer encoder + GPT-2 decoder
Base checkpoint	nlpconnect/vit-gpt2-image-captioning
Browser baseline	Xenova/vit-gpt2-image-captioning
Published deployment model	Project quantized ONNX Vision Encoder-Decoder package
Decoding methods	Greedy decoding and beam search
Evaluation	BLEU-1, BLEU-4, METEOR, ROUGE-L, CIDEr, CLIPScore, diversity, repetition, latency
Deployment	Hugging Face Model Hub + free Hugging Face Static Space
Runtime model	Browser-side ONNX inference with no paid Python backend

---

Dataset and Data Preparation

The training and evaluation workflow uses paired images and natural-language captions, separated into training, validation, and held-out test partitions.

The project preparation pipeline is responsible for:

• Reading image paths and associated captions
• Validating missing or unreadable images
• Normalizing caption text
• Preserving reproducible dataset splits
• Converting images to RGB
• Applying the selected image processor
• Tokenizing caption targets
• Building attention masks
• Padding and truncating caption sequences
• Preparing batches for Vision Encoder-Decoder training
• Saving reusable metadata and evaluation artifacts

Large raw datasets, generated caches, virtual environments, local checkpoints, and temporary training files are excluded from normal Git tracking. The repository focuses on source code, configuration, notebooks, evaluation artifacts, deployment assets, and reproducible documentation.

---

Tools and Technologies

Area	Technology
Languages	Python, JavaScript, HTML, CSS
Deep learning	PyTorch
Transformer library	Hugging Face Transformers
Vision encoder	Vision Transformer
Text decoder	GPT-2
Architecture wrapper	VisionEncoderDecoderModel
Image processing	PIL and Hugging Face image processing
Data analysis	pandas, NumPy
Caption evaluation	BLEU, METEOR, ROUGE-L, CIDEr, CLIPScore
Model export	ONNX, opset 18
Browser inference	ONNX Runtime Web and Transformers.js
Interface	Static HTML, CSS, and JavaScript
Testing	pytest and project-structure validation
Model hosting	Hugging Face Model Hub
Application hosting	Hugging Face Static Spaces
Development hardware	NVIDIA GeForce RTX 5090
Python environment	Python 3.12

---

End-to-End Project Workflow

Paired images and reference captions
                │
                ▼
       Data validation and splitting
                │
                ▼
   RGB conversion and image preprocessing
                │
                ▼
      Caption cleaning and tokenization
                │
                ▼
       ViT image-feature encoding
                │
                ▼
 GPT-2 autoregressive caption generation
                │
                ▼
   GPU fine-tuning and checkpoint selection
                │
                ▼
 Pretrained vs fine-tuned test evaluation
                │
                ▼
BLEU, METEOR, ROUGE-L, CIDEr, CLIPScore
                │
                ▼
 Diversity, repetition, latency, error review
                │
                ▼
     ONNX opset-18 model conversion
                │
                ▼
 Dynamic quantization and parity validation
                │
                ▼
     Hugging Face Model Hub publishing
                │
                ▼
 Client-side Static Space deployment

---

Vision Encoder-Decoder Architecture

The project combines two pretrained Transformer components:

• A Vision Transformer encoder converts the input image into contextual visual representations.
• A GPT-2 decoder generates the caption autoregressively, one token at a time.
• Cross-attention allows the decoder to condition each generated token on the encoder's image features.

Input image
    │
    ▼
Resize, normalize, and batch
    │
    ▼
Vision Transformer encoder
    │
    ▼
Contextual image embeddings
    │
    ▼
GPT-2 decoder with cross-attention
    │
    ├── Previous caption tokens
    └── Encoded image features
    │
    ▼
Next-token probability distribution
    │
    ▼
Greedy decoding or beam search
    │
    ▼
Generated image caption

Selected base checkpoint

nlpconnect/vit-gpt2-image-captioning

The project performs additional training and evaluation rather than presenting the original checkpoint as project-created work.

---

Fine-Tuning Strategy

The training workflow uses the RTX 5090 GPU and a reproducible Vision Encoder-Decoder pipeline.

Core training behavior includes:

• Image preprocessing through the model-compatible processor
• Caption tokenization through the GPT-2 tokenizer
• Teacher-forced autoregressive training
• Validation during model development
• Checkpoint and metadata preservation
• GPU-aware batch processing
• Reproducible experiment configuration
• Held-out test evaluation
• Separate pretrained and fine-tuned generation exports

The fine-tuned PyTorch checkpoint is approximately 917 MB when serialized. Large training checkpoints are intentionally excluded from GitHub and are handled separately from the source repository.

---

Decoding Strategies

Greedy decoding

Greedy decoding selects the highest-probability next token at every generation step.

Advantages:

• Simple
• Deterministic
• Fast
• Easy to inspect

Trade-off:

• A locally optimal token choice may produce a weaker overall caption.

Beam search

Beam search maintains several candidate token sequences and keeps the strongest cumulative paths.

Advantages:

• Explores multiple caption candidates
• Can improve sequence-level output
• Demonstrates practical autoregressive search

Trade-offs:

• Slower than greedy decoding
• Can favor safe or generic captions
• Beam width and length penalties affect output behavior

---

Model Results

The final comparison uses 1,000 held-out test examples for both the pretrained and project-fine-tuned checkpoints.

Metric	Pretrained checkpoint	Project-fine-tuned checkpoint	Difference
BLEU-1	0.5779	0.4350	-0.1429
BLEU-4	0.1366	0.1078	-0.0288
METEOR	0.3337	0.4050	+0.0713
ROUGE-L	0.3759	0.3041	-0.0718
CIDEr	0.2336	0.0595	-0.1741
CLIPScore	0.6601	0.7211	+0.0610
Mean GPU latency	0.1163 s	0.2762 s	+0.1599 s
Unique-caption ratio	Not reported	1.0000	—
Average repetition	Not reported	0.1463	—

Honest interpretation

The fine-tuned checkpoint is not universally superior across every metric.

It improved:

• METEOR
• CLIPScore
• Lexical uniqueness
• Semantic alignment with image content

It regressed on:

• BLEU-1
• BLEU-4
• ROUGE-L
• CIDEr
• Mean inference latency

The fine-tuned model is approximately 2.37 times slower than the pretrained checkpoint in the reported GPU latency comparison.

This mixed result is important. It shows that fine-tuning changed model behavior and improved semantic similarity and diversity, but did not improve every n-gram-based benchmark. The project reports the trade-off rather than hiding unfavorable metrics.

---

Why Multiple Caption Metrics Matter

• BLEU-1 measures unigram overlap between generated and reference captions.
• BLEU-4 measures stricter multi-token phrase overlap.
• METEOR considers token alignment and gives partial credit for related wording.
• ROUGE-L measures longest-common-subsequence overlap.
• CIDEr measures consensus against multiple human captions using weighted n-grams.
• CLIPScore measures image-text semantic alignment through a joint vision-language embedding space.
• Unique-caption ratio helps reveal output collapse or repeated generation.
• Repetition statistics help identify loops and redundant phrases.
• Latency exposes the deployment cost of model complexity and decoding strategy.

Image captioning can produce a semantically reasonable sentence that differs from reference wording. For that reason, no single metric provides a complete evaluation.

---

ONNX Export and Browser Deployment

The browser deployment uses ONNX graphs exported with opset 18.

The generated browser package includes:

Artifact	Approximate size
Quantized encoder	87 MB
Quantized decoder	155 MB
Quantized decoder with past	155 MB
Quantized merged decoder	614 MB

The complete published browser-model package is approximately 1.01 GB.

ONNX parity validation

PyTorch-to-ONNX parity was checked on three representative images:

• road.png
• seaside.png
• workspace.png

The parity test reported:

• Exact caption match on all three samples
• Mean token Jaccard similarity: 1.0
• Overall parity status: Passed

This verifies that the exported browser artifacts reproduced the checked PyTorch generation outputs for the selected parity samples.

---

Live Browser Demo

The deployed application performs model execution inside the visitor's browser. It does not require a paid GPU server or Python inference backend.

Live Application

Published ONNX Model

Application Overview

Live Project 02 interface showing the browser-based image-captioning workflow, sample-image selection, model options, decoding controls, and deployment links.

Fine-Tuned Greedy Caption

Project-fine-tuned caption generation using greedy decoding on a selected image.

Beam-Search Captioning

Beam-search generation demonstrating sequence-level search beyond single-path greedy decoding.

Evaluation Details

Pretrained-versus-fine-tuned evaluation summary covering caption quality, semantic alignment, diversity, and latency trade-offs.

---

Browser Inference Workflow

User selects a sample or uploads an image
                  │
                  ▼
 Browser validates file type and image content
                  │
                  ▼
       Image is resized and normalized
                  │
                  ▼
 ONNX Vision Transformer encoder executes
                  │
                  ▼
 Encoded image representation is produced
                  │
                  ▼
 ONNX GPT-2 decoder generates token logits
                  │
                  ▼
 Greedy or beam-search decoding is applied
                  │
                  ▼
 Tokens are converted into a readable caption
                  │
                  ▼
 Caption, timing, trace, and comparison details

The first model load may take longer because the browser must download and cache large ONNX files.

---

Core Python Model Versus Live Browser Runtime

Component	Evaluated Python pipeline	Live Static Space
Framework	PyTorch and Hugging Face Transformers	ONNX Runtime Web / Transformers.js
Architecture	ViT encoder + GPT-2 decoder	ViT encoder + GPT-2 decoder
Fine-tuned artifact	Local project checkpoint	Quantized ONNX deployment package
Baseline	Original pretrained checkpoint	Browser-compatible pretrained baseline
Runtime	Local GPU or CPU	Visitor's browser
Primary purpose	Training, testing, benchmarking, export	Free interactive portfolio demonstration
Server-side inference	Optional locally	None
Paid compute required	No for source use; local hardware required	No

The Static Space performs genuine client-side model inference. Browser speed and compatibility depend on the visitor's device, memory, browser engine, network connection, WebAssembly support, and WebGPU availability.

---

Failure Analysis

The project includes structured failure analysis rather than relying only on aggregate metrics.

Common image-captioning failure patterns include:

• Generic captions that omit distinguishing details
• Incorrect object recognition
• Missing secondary objects
• Incorrect action descriptions
• Wrong object counts
• Incorrect colors or attributes
• Weak spatial relationships
• Repetition
• Premature caption termination
• Overly long generation
• Captions that are linguistically fluent but visually unsupported

The generated report is stored at:

outputs/failure_analysis.md

The final summarized project findings are stored at:

outputs/RESULTS_SUMMARY.md

---

Model and Evaluation Artifacts

Artifact	Purpose
outputs/fine_tuned_metrics.json	Fine-tuned caption-evaluation metrics
outputs/pretrained_vs_finetuned_comparison.csv	Direct model comparison
outputs/fine_tuned_generated_caption_examples.csv	Generated fine-tuned captions
outputs/fine_tuned_clipscore_results.json	Per-example CLIPScore results
outputs/onnx_parity_results.json	PyTorch-to-ONNX parity validation
outputs/failure_analysis.md	Structured failure review
outputs/RESULTS_SUMMARY.md	Final human-readable project summary
web/sample_outputs/	Browser-ready evaluation and trace artifacts
web/sample_images/	Safe images used in the public demonstration
MODEL_CARD.md	Published model documentation
README_HUGGINGFACE_STATIC.md	Static Space metadata and documentation

Large model checkpoints, intermediate ONNX exports, training runs, caches, virtual environments, and upload staging folders are excluded from normal Git tracking.

---

Run the Python Project Locally

1. Open the project

cd /d D:\AI-Training\encoder-decoder-projects\02-image-caption-generation-vision-encoder-decoder

2. Create and activate a virtual environment

Windows

py -3.12 -m venv .venv
.venv\Scripts\activate

macOS / Linux

python3.12 -m venv .venv
source .venv/bin/activate

3. Install dependencies

Install a CUDA-enabled PyTorch build appropriate for the machine, then run:

python -m pip install --upgrade pip
python -m pip install -r requirements.txt

4. Validate the project structure

python scripts/validate_project.py

5. Run the test suite

python -m pytest tests -q

The completed project validation reported 18 passing tests.

---

Run the Static Website Locally

From the project root:

python -m http.server 8000

Open:

http://127.0.0.1:8000/web/

The first model inference may take longer because the browser downloads and caches the required model artifacts.

---

Deployment

GitHub

• Repository: unit-mole/encoder-decoder-projects
• Branch: main
• Project folder: 02-image-caption-generation-vision-encoder-decoder/

Hugging Face Model Hub

• Model repository: anmol-unitmole/image-caption-generation-vision-encoder-decoder-model
• Model type: Vision Encoder-Decoder for image captioning
• Base model: nlpconnect/vit-gpt2-image-captioning
• Deployment format: Quantized ONNX
• ONNX opset: 18
• Processor and tokenizer metadata: Included
• Model card: Included

Hugging Face Static Space

• Space: anmol-unitmole/image-caption-generation-vision-encoder-decoder
• SDK: Static
• Browser inference: ONNX Runtime Web / Transformers.js
• Server-side model compute: None
• Python backend: Not required
• Cost: Free static hosting

---

Project Structure

encoder-decoder-projects/
├── .github/
│   └── workflows/
│       └── project workflows
│
└── 02-image-caption-generation-vision-encoder-decoder/
    ├── configs/
    │   └── training and evaluation configuration
    ├── data/
    │   └── data preparation and metadata
    ├── images/
    │   ├── 01-application-overview.png
    │   ├── 02-finetuned-greedy-caption.png
    │   ├── 03-beam-search-results.png
    │   └── Evaluation_Details.png
    ├── models/
    │   ├── fine_tuned_model/          # local / ignored
    │   ├── hub_browser_model/         # upload staging / ignored
    │   └── onnx/                      # generated / ignored
    ├── notebooks/
    │   └── six project notebooks
    ├── outputs/
    │   ├── failure_analysis.md
    │   ├── RESULTS_SUMMARY.md
    │   ├── evaluation metrics
    │   ├── caption examples
    │   └── parity results
    ├── scripts/
    │   ├── configure_deployment.py
    │   ├── sync_web_artifacts.py
    │   ├── validate_project.py
    │   └── training, evaluation, export, and deployment utilities
    ├── src/
    │   └── dataset, model, training, evaluation, and inference modules
    ├── tests/
    │   └── project test suite
    ├── web/
    │   ├── sample_images/
    │   ├── sample_outputs/
    │   ├── app.js
    │   ├── attention_view.js
    │   ├── caption_inference.js
    │   ├── decoding.js
    │   ├── image_preprocessing.js
    │   ├── index.html
    │   ├── metadata.json
    │   ├── style.css
    │   └── token_generation_view.js
    ├── MODEL_CARD.md
    ├── README_HUGGINGFACE_STATIC.md
    ├── README.md
    └── requirements.txt

---

Testing and Validation

The project includes:

• Dataset-loader validation
• Model-configuration validation
• Caption-generation validation
• Decoding validation
• Evaluation-output validation
• ONNX export checks
• PyTorch-to-ONNX parity checks
• Static-file validation
• Sample-image validation
• Browser metadata validation
• Project-structure validation

Final local test status:

18 passed

Final project validator status:

passed

---

Limitations

• Caption quality varies significantly across image types and visual complexity.
• The model can hallucinate objects or relationships.
• Small, occluded, rare, or visually ambiguous objects may be missed.
• Text embedded inside images is not reliably transcribed.
• The model does not provide verified factual descriptions.
• Fine-tuning improved some semantic metrics but reduced several n-gram metrics.
• The fine-tuned model is slower than the pretrained checkpoint in the reported benchmark.
• Beam search increases inference cost.
• Browser inference requires downloading large ONNX files.
• Initial loading may be slow on limited connections.
• Mobile devices and low-memory systems may struggle with the deployment package.
• Browser performance varies by CPU, memory, WebAssembly, WebGPU, and browser implementation.
• Quantization can alter numerical behavior, although the checked parity samples passed.
• The three parity images do not prove equivalence for every possible image.
• The project has not been validated for production or safety-critical use.

---

Future Improvements

• Evaluate additional public image-captioning datasets.
• Add stronger pretrained vision-language architectures.
• Compare ViT-GPT-2 with BLIP, BLIP-2, GIT, and modern multimodal instruction models.
• Add calibrated sequence confidence.
• Add object-level grounding and region visualization.
• Add semantic beam reranking using CLIPScore.
• Add top-k and nucleus-sampling controls.
• Add repetition penalties and stronger length normalization.
• Improve rare-object and attribute recognition.
• Add multilingual caption generation.
• Add accessibility-focused caption review.
• Add automated qualitative error clustering.
• Add larger browser-performance benchmarks.
• Explore lower-bit quantization and model distillation.
• Reduce the merged decoder download size.
• Add browser integration and accessibility tests.
• Add a dedicated Project 02 GitHub Actions workflow.

---

Skills Demonstrated

• Computer vision
• Natural-language generation
• Multimodal machine learning
• Vision Transformer architecture
• GPT-2 autoregressive decoding
• Vision Encoder-Decoder integration
• Image preprocessing
• Caption tokenization
• GPU fine-tuning
• Greedy decoding
• Beam search
• BLEU evaluation
• METEOR evaluation
• ROUGE-L evaluation
• CIDEr evaluation
• CLIPScore evaluation
• Caption-diversity analysis
• Repetition analysis
• Latency benchmarking
• Failure analysis
• PyTorch model development
• ONNX export
• Dynamic quantization
• PyTorch-to-ONNX parity testing
• Transformers.js
• ONNX Runtime Web
• Static web application development
• Hugging Face Model Hub publishing
• Hugging Face Static Space deployment
• Model-card documentation
• Responsible AI communication
• Portfolio-focused ML engineering

---

Portfolio Positioning

One-line description: End-to-end ViT-GPT-2 image-captioning system featuring GPU fine-tuning, pretrained-versus-fine-tuned evaluation, greedy and beam-search decoding, ONNX quantization, parity validation, model publishing, and free browser deployment.

Pinned repository description: Professional multimodal AI portfolio project combining a Vision Transformer encoder and GPT-2 decoder, caption-quality evaluation, CLIPScore analysis, failure analysis, quantized ONNX browser inference, Hugging Face Model Hub publishing, and a live Static Space.

This project connects naturally to Data Science, Machine Learning, Applied AI, Computer Vision, NLP, multimodal AI, analytics engineering, and model-deployment roles.

---

Author

Anmol Tripathi

Quality Data Scientist building a professional portfolio in Data Science, Machine Learning, Applied AI, Natural Language Processing, Computer Vision, Transformer Models, Generative AI, Analytics Engineering, and Quality Analytics.