# 1. Abstract This is a Deep Learning Model for generating captions for images. It uses techniques from Computer Vision and Natural Language Processing. Some examples of images from test dataset and the captions generated by the model are shown below. ![ezgif-1-ba79ce21f2.gif](ezgif-1-ba79ce21f2.gif) # 2. Introduction Deep Learning and Neural Networks have found profound applications in both NLP and Computer Vision. Before the Deep Learning era, statistical and Machine Learning techniques were commonly used for these tasks, especially in NLP. Neural Networks however have now proven to be powerful techniques, especially for more complex tasks. With the increase in size of available datasets and efficient computational tools, Deep Learning is being throughly researched on and applied in an increasing number of areas.Image captioning is the process of taking an image and generating a caption that accurately describes the scene. This is a difficult task for neural networks because it requires understanding both natural language and computer vision. It is the intersection between NLP and Computer Vision. ![Screenshot_89.png](Screenshot_89.png) The purpose of your image captioning project is to develop a system that can generate descriptive captions for images. The primary objective is to automate the process of generating accurate and meaningful textual descriptions that capture the visual content and context of the images. ![generated.gif](generated.gif) # 3. Related work 3.1 Dataset This project uses the [Flickr 8K](https://www.kaggle.com/datasets/adityajn105/flickr30k) dataset for training the model. This can be downloaded from here. It contains 8000 images, most of them featuring people and animals in a state of action.Though i have used two more dataset like [Flickr 8K](https://www.kaggle.com/datasets/adityajn105/flickr30k) and [Microsoft CoCo](https://www.kaggle.com/datasets/awsaf49/coco-2017-dataset) but flickr 8k comparatively doing well from among datasets.Here Each image is provided with five different captions describing the entities and events depicted in the image. Different captions of the same image tend to focus on different aspects of the scene, or use different linguistic constructions. This ensures that there is enough linguistic variety in the description of the images. # 4. Used Model In this project i used several models like vgg16 with lstm,resnet50 with lstm, inception v3 with lstm but considering everything i deployed vgg16 with lstm for my convenience the Vgg16 architecture for obtaining the image features. VGG networks, standing for Visual Geometry Group networks, have played a pivotal role in the realm of Computer Vision. Their significance was highlighted when they secured victory in the prestigious ImageNet Challenge. VGG networks demonstrated a groundbreaking concept—how even deep neural networks (VGG16 and VGG19 are examples) can be effectively trained, transcending the limitations of the vanishing gradient problem. The hallmark of VGG networks is their simplicity and uniformity. Unlike the original ResNet with its 152 layers, VGG networks feature a straightforward and easily understandable architecture. This simplicity, combined with their effectiveness, makes them a popular choice for various computer vision tasks. In the world of Transfer Learning, VGG networks shine brightly. Keras, a widely-used deep learning framework, includes pre-trained VGG models along with weights fine-tuned on the expansive ImageNet dataset. This availability has made VGG networks a go-to choice for researchers and practitioners in the field. Since we only need this network for getting the image feature vectors, so we remove the last layer. ![Encoder-Decoder-Framework-for-Image-captioning-3.png](Encoder-Decoder-Framework-for-Image-captioning-3.png) # 5. Experiments Models | Accuracy | BLEU-1 | BLEU-2 --- | --- | --- | --- Vgg 16 with Lstm | 0.5125 | 0.540437 | 0.316454 Resnet 50 with Lstm | 0.5522 | 0.538153 | 0.321559 InceptionV3 with Lstm | 0.5012 | 0.545291 | 0.323035 # 6. Prediced Results Image | Caption --- | --- ![3071676551_a65741e372.jpg](3071676551_a65741e372.jpg) | **Generated Caption:** Surfer rides the waves. ![e98c741a23421241f7ce5cb399ead8762ec71f2b74e1e58ad957eae4.jpg](e98c741a23421241f7ce5cb399ead8762ec71f2b74e1e58ad957eae4.jpg)|**Generated Caption:** Woman in green shirt and glasses is climbing large rock. ![78c65793b4613ed5d95686c90cad4c6856e018f3727b139220b38223.jpg](78c65793b4613ed5d95686c90cad4c6856e018f3727b139220b38223.jpg) |**Generated Caption:** Dog jumps over hurdle. ![961b377b96dbf7cbe2729e0552d2bd7e720c7bc16bf761fd7b53efeb.jpg](961b377b96dbf7cbe2729e0552d2bd7e720c7bc16bf761fd7b53efeb.jpg)| **Generated Caption:** Man in blue jacket snowboarding. ![738020db6a97154799_0x0.jpg](738020db6a97154799_0x0.jpg) | **Generated Caption:** Basketball player dribbles the ball. # 7. Discussion 7.1 Used Frameworks, Libraries & Languages

Keras
Tensorflow
Python3
Numpy
Matplotlib
pickle

7.2 Deployment I made a web application using streamlit framework. This web application is hosted and deployed on shere.streamlit platforms: [share.streamlit.io](https://imagecaptiongenerator-cjmnhj4scsrxheqxtsmune.streamlit.app/) and the implementation can be found in this [Github repository](https://github.com/riad5089/Image_Caption_Generator). 7.3 Demo Video https://github.com/riad5089/Image_Caption_Generator/assets/93583569/270f348f-404a-4823-b014-bed14be0d5b1 # 8. Conclusion 8.1 Limitations This image caption generator application shows promise but currently faces challenges in generating accurate captions due to factors such as limited computational resources, a relatively small dataset, occasional GPU issues, and memory constraints. However, I am committed to making continuous improvements and enhancements to enhance its performance in the future as well as i made this project using a LLM model called Gemini-Pro-Vision you can check out this [Github repository](https://github.com/riad5089/Image-Captioning-Web-App-with-Gemini-Pro-Vision) which has more accurate prediction. If you have any suggestions regarding this feel free to send me pull request. # 9. Acknowledgements

I referred many articles & research papers while working on this project. Some of them are listed below-

https://www.analyticsvidhya.com/blog/2021/12/step-by-step-guide-to-build-image-caption-generator-using-deep-learning/
https://data-flair.training/blogs/python-based-project-image-caption-generator-cnn/
https://towardsdatascience.com/understanding-and-coding-a-resnet-in-keras-446d7ff84d33
https://www.hackersrealm.net/post/image-caption-generator-using-python

# 10. Appendix 10.1 Code Snippets Include key code snippets or scripts that are integral to my project over [here](https://github.com/riad5089/Image_Caption_Generator/tree/main/Notebooks), such as: Preprocessing the dataset. Model architecture and training process. Evaluation metrics and calculation of BLEU scores. Deployment scripts for the Streamlit application. 10.2 Dataset Details Provide a detailed description of the datasets used (Flickr 8K, Flickr 30K, and MS COCO): Number of images. Source and link to download. Sample captions for a few images. 10.3 Hyperparameters Document the hyperparameters used during training, such as: Learning rate. Batch size. Epochs. Optimizer. 10.4 Results Logs Provide raw logs or outputs from model training, validation, and testing phases, such as: Training accuracy and loss plots. BLEU score logs. 10.5 Challenges and Limitations List the challenges encountered, such as: GPU or memory constraints. Dataset size limitations. Model overfitting or underfitting issues. 10.6 Screenshots of the Application Add screenshots of the deployed web application, demonstrating its features and functionality. 10.9 Acknowledgments Include a detailed acknowledgment.