Abstract

This publication presents a web-based application designed to compare various neural networks for image segmentation. By enabling side-by-side analysis of segmentation outputs, the platform helps researchers, developers, and practitioners determine the most suitable model for their specific requirements. The application is structured around a modern technology stack, including an Angular frontend, Node.js with Express.js for backend services, and Python for executing AI models, all underpinned by an SQLite database for efficient data management.

Key features include intuitive image management with category-based organization, interactive overlay animations for visually comparing segmentation masks, and a flexible design that supports seamless addition of new categories and images.The system’s primary use case—verifying whether two images represent the same individual—demonstrates its real-world applicability, notably in projects like ADRIAN. Through a well-documented, extensible architecture, this platform aims to streamline the otherwise complex process of selecting and deploying optimal image segmentation models. Future enhancements will expand its functionality by integrating image origin tracking, tag management, additional neural network segmentation models, and improved user interactions. Overall, this application offers a powerful, user-friendly interface for exploring the strengths and limitations of different segmentation approaches in varied use-case scenarios.

Here’s a sneak peek of the finished solution:

This is the link to this project on Github: https://github.com/vika-v-v/neural-networks-for-image-segmentation.

Introduction and Motivation

Image segmentation is a fundamental process in computer vision, enabling systems to identify and isolate distinct regions or objects within an image. It underpins a wide variety of applications, from medical imaging to autonomous vehicles and security systems. Despite its critical importance, selecting the most suitable neural network for a specific segmentation task can be challenging. Different models may excel in accuracy, speed, or resource requirements, and there is no one-size-fits-all solution.

This publication introduces a web-based application that simplifies the process of comparing multiple neural networks for image segmentation. Through an intuitive interface, users can upload their own images, organize them into categories, and view side-by-side segmentation results for various AI models. By providing a centralized, user-friendly platform, this application accelerates the decision-making process—particularly when determining whether two images depict the same individual, a use case pivotal to projects like ADRIAN.

The motivation behind this project is to provide a robust yet accessible solution for anyone looking to determine which neural network best meets their image segmentation requirements. Through a blend of interactive features, category-based organization, and a future-oriented design, the platform offers a valuable resource for both research and practical deployment scenarios.

Objectives

1. Enable Effective Comparison of Neural Networks
   Provide a straightforward mechanism for side-by-side analysis of segmentation results, helping users quickly identify which model(s) best suit their specific use cases or data requirements.


2. Offer a User-Friendly and Intuitive Interface
   Design an accessible frontend that streamlines image upload, organization, and comparison. This lowers the barrier for both technical and non-technical users to experiment with and evaluate AI models.


3. Ensure Flexibility and Extensibility
   Build the system with a modular architecture that simplifies the addition of new neural networks, categories, and features as the field of deep learning continues to evolve.


4. Foster Community Contributions
   Encourage open-source involvement by providing comprehensive documentation and guidelines for adding models, creating new features, and refining the platform based on user feedback.

Approach

The project was developed in a series of clearly defined steps, each building upon the previous one to ensure an efficient workflow:

Problem Analysis and Technology Selection

The process began with a thorough requirements analysis to identify both functional and non-functional needs. These included:

1. Image Upload and Management:
   • The system shall provide functionality for users to upload images in common formats (e.g., JPG, PNG).
   • The system shall allow users to categorize images (e.g., “faces,” “clothes,” etc.) for organizational purposes.
   • The system shall enable users to add, delete, and edit metadata such as tags, origin, and notes.
   • The user interface should be intuitive, with clear navigation and actionable feedback for each interaction.


2. Neural Network Selection and Comparison:
   • The system shall display results from multiple models in a side-by-side or overlay format for easier comparison.
   • The system shall automatically execute the selected models on user-uploaded images and retrieve the segmentation outputs.
   • The system should process model inference within a reasonable response time, ensuring the user experience is not hindered by excessive wait times.


3. Visualization and Analysis:
   • The system shall present segmentation overlays or masks on the original images.
   • The system shall allow users to view different segmentation layers or masks to compare results more effectively.

Based on these requirements, such technologies were selected:

• Angular (frontend),
• Node.js with Express.js (backend),
• Python (for AI model execution),
• SQLite (database).

Because the requirements did not specify the exact models to be used, three representative models were chosen to showcase differing yet somewhat overlapping capabilities. This allows for meaningful comparisons while illustrating unique strengths and use cases:

• jonathandinu/face-parsing - Specializes in segmenting facial features, extracting parts of the face for tasks such as facial recognition or facial analysis.
• mattmdjaga/segformer_b2_clothes - Focuses on clothing and accessories, making it relevant for fashion analysis or virtual try-on applications.
• nvidia/segformer-b0-finetuned-ade-512-512 - A more general-purpose model that segments a wide range of objects within an image, enabling comprehensive scene understanding.

Architecture Development

Next, the software architecture was designed to define how the frontend, backend, AI layer, and database would interact.

As illustrated in the diagram, the system is composed of a frontend with a homepage and a comparison page, as well as a backend featuring a Node.js server. The Node.js server handles database operations—such as storing and retrieving images—and mediates communication with a Python server. When requested by the Node.js server, the Python server processes images using a specific AI model.

From the figure, you can see that the exposed APIs fall into two main categories:

• Model-Specific Data – e.g., segments generated by a particular model.
• Image and Category Management – e.g., endpoints like imageById for accessing or updating images and categories.

Database Structure Development

Building on the architectural blueprint, the database schema was designed with a focus on managing and categorizing uploaded images and storing essential metadata.

Database Population

Once the database setup was complete, test data was added to validate the system in practice. This early data integration helped to ensure correct data storage and retrieval. Later, real data was added, such as images and associated categories, to provide a quick overview without the user having to add all the data themselves. There is also an initDatabase.js script that imports all the data after the user has installed the project from Github (Getting Started guide: https://github.com/vika-v-v/neural-networks-for-image-segmentation/blob/master/docu/GETTING_STARTED.md).

Frontend Development with Mock API Endpoints

Key frontend components were initially developed using mock API endpoints. This allowed for early testing of user interface elements, interactions, and workflows, even before a fully functional backend was in place.

Real API Endpoints and Backend Development

In parallel with the frontend development, the actual backend API endpoints were implemented. This involved creating the necessary functions for running AI models, storing segmentation results, and facilitating communication between the frontend and the database.

Testing and Documentation

Finally, comprehensive testing was performed to confirm the stability and correctness of the entire system. This included both automated tests and manual end-to-end testing to verify how all components work together. The findings were thoroughly documented, creating a solid foundation for future enhancements and the integration of additional AI models.

Results and demo

By the end of the development cycle, the project met nearly all of its stated objectives and requirements, resulting in a functional, user-friendly platform for comparing neural networks in image segmentation. The full code is available on Github: https://github.com/vika-v-v/neural-networks-for-image-segmentation. This is a demo video:

Below is an overview of how the final system aligns with each objective:

1. Enable Effective Comparison of Neural Networks
   • Achieved: Users can upload images, select from multiple AI models, and view the segmentation outputs side by side. This setup makes it straightforward to assess which model best fits a particular use case.

2. Offer a User-Friendly and Intuitive Interface
   • Achieved: The Angular-based frontend provides a simple workflow for image upload and category organization. Users can easily navigate to the comparison page and review segmentation results—all without needing in-depth technical knowledge.

3. Ensure Flexibility and Extensibility
   • Achieved: A modular architecture—split into a Node.js backend, a Python-based AI layer, and a SQLite database—allows new models to be integrated with minimal disruption. This design choice also simplifies the addition of new features or categories in the future.

4. Foster Community Contributions
   • Achieved: The repository includes detailed documentation (e.g., GETTING_STARTED, ADDING_NEURAL_NETWORK) that outlines how to add new models or contribute frontend and backend features. This transparency and guidance encourage open-source collaboration. The technical documentation is to be found on Github: https://github.com/vika-v-v/neural-networks-for-image-segmentation.

Requirements Fulfilled

• Neural Network Comparison:
  The platform supports multiple segmentation models, automatically executes them, and displays their results side by side for quick evaluation.

• Visualization and Analysis:
  The system overlays segmented masks on the original images and shows different segmentation layers, providing clear insights into how each model interprets the image.

• Image Upload and Management:
  Users can upload JPEG and PNG images into the system, where they are stored in the database and organized by category. There is also an option to generate the person from ThisPersonDoesntExist:

You can remove the image or change the categories later, by clicking the corresponding symbol on the image in the first column.

• Category Upload and Management:
  Users can create, edit and delete categories. Each category can have undercategories. For example, there could be a category "Age" with undercategories "Child", "Teen", "Adult" and "Elderly". For each undercategory, any number of images can be selected from those already added. To add a new image to an undercategory, create a new image by clicking on "Add image" and then select the desired undercategory in the second step.

• Performance and Responsiveness:
  While specific metrics may vary, tests indicate that model inference runs within a reasonable time frame, maintaining a fluid user experience.

Not Yet Implemented

• Metadata Management (Tags, Origin, Notes):
  Although planned, the feature that allows users to add, delete, and edit metadata such as tags, origin, and notes is still under development. Currently, placeholders exist in the interface, but saving or editing these fields is not yet active.

---

Overall, this project delivers a robust foundation for comparing neural networks in image segmentation, fulfilling its core objectives and most stated requirements. As development continues, planned enhancements—such as metadata management—will further extend the platform’s capabilities, offering an even richer environment for researchers, developers, and open-source contributors.

Encountered Problems

Inconsistent Segmentation Model Responses

Different segmentation models produce outputs in varying formats. Some return an array of pixels indicating, via a “true” value, whether each pixel belongs to a particular segment, whereas others provide a Base64-encoded image for each segment. To address this, a standardized response structure had to be defined for the frontend. This involved creating separate handlers for each model to transform its unique response into the agreed-upon format.

Real-Time Segment Indication

Implementing real-time highlighting of segments as the user hovers over the image posed challenges on both the frontend and backend:

• Frontend:
  Angular does not natively support hovering parts of an image to dynamically reveal segments. Simply stacking segment images on top of one another proved difficult because it was not clear which segment was being hovered. To solve this, a canvas was used where all segments are “drawn” manually. TypeScript logic then determines the pixel coordinates of the hover event, checks each segment’s data, and highlights (or hides) everything else accordingly.

• Backend:
  Processing segments on the fly took too long, making real-time interaction impractical. To optimize performance, a lazy-loading mechanism was implemented. Once a segment is processed, it is stored in the database (acting as a cache). Subsequent requests for the same image and segment can then be served quickly without reprocessing.

Python-Only Libraries

Initially, the project relied solely on Node.js for server operations. While CRUD functionalities ran smoothly, certain critical tools for image processing were available only in Python. As a result, a separate Python backend was introduced specifically for handling segmentation tasks, leveraging the necessary libraries while the Node.js server continued to manage database operations and API endpoints.

Non-Technical Considerations

Several non-technical issues also arose, including:

• Image Licensing:
  Only royalty-free images could be used to avoid legal complications.
• Ethical Concerns:
  Categorizing people by characteristics such as gender and age may raise moral or privacy questions. In this project, these categories are intended solely for demonstration purposes and do not imply any real-world system of classification or judgment.

Future Development

1. Adding More Neural Network Models
   Currently, the platform supports only three models—focusing on face, clothing, and general object segmentation—which do not offer a comprehensive overview of all available segmentation approaches. Expanding the roster to include additional, specialized models (e.g., medical imaging, satellite imagery) would greatly enhance its utility. Given the system’s modular architecture and clear documentation, integrating new models is relatively straightforward, yet would still provide significant added value.

2. Automated Category Creation
   A potential improvement is to introduce a neural network that suggests or automatically creates categories for newly added images. This feature would streamline user experience by reducing manual categorization, particularly in large-scale or dynamic datasets.

3. Search Functionality
   Implementing a robust search tool for images, categories, or even neural network models would help users quickly find the data they need. This functionality becomes increasingly beneficial as more models and images are integrated into the platform.

4. Beyond Segmentation Comparisons
   While the current UI excels at comparing image segmentation models, the overall structure can be extended to support other AI tasks. For instance, the application could host comparison tools for image generation, text summarization, or any additional deep learning approaches. This would transform the platform into a more general-purpose environment for evaluating and understanding a wide range of AI capabilities.

Conclusion

This publication has presented the development and implementation of a web-based application designed to simplify the comparison of neural networks for image segmentation. By providing a modular architecture—comprising an Angular frontend, a Node.js backend, and a Python-based AI layer—this platform successfully meets its core objectives of enabling effective model comparison, delivering an intuitive user experience, and fostering open-source contributions. The comparison results highlight each model’s strengths in tasks such as facial segmentation, clothes detection, or generalized object classification, underscoring the platform’s versatility.

Although the current system offers a solid foundation for segmentation model analysis, there remain exciting avenues for future development. These include expanding the library of neural networks, incorporating automated category creation, and adding more sophisticated search features. More broadly, the platform’s extensible design opens the door for comparing other AI models beyond segmentation, such as image generation and text summarization, making it a valuable, evolving resource for researchers, developers, and practitioners. Overall, the application stands as a user-friendly and scalable solution for those seeking to explore, evaluate, and deploy image segmentation models across a range of real-world scenarios.