A critique of an existing AI medication-identification app, paired with a team prototype: an AI scanner and web interface that classifies medicines as approved, unapproved, or illegal at airport security checkpoints.
Airports increasingly lean on AI to speed up and strengthen security — but the app we chose to critique, Smart Pill Identifier, correctly recognised only two of five similar-looking pills a team member scanned. We proposed a narrower, purpose-built alternative instead: an AI-powered scanning device with a database of prohibited medicines, built to remove the pressure of accurately labelling medication from travellers and hand that judgement to airport officials with a trained classifier behind them.
In the current travel landscape, facilitating a seamless flow of goods and people through airports is essential. However, alongside the benefits of increased mobility comes significant challenges, particularly in safeguarding against the trafficking of illicit substances. Over the last few years, integration of AI technology into airport security systems has gained prominence — but it also brings widespread public concerns, particularly regarding privacy, ethics, and cyber security1.
We focused on AI technology that swiftly identifies medications through visual recognition. The application we chose to critique for our prototype is Smart Pill Identifier, available on the Google Play Store and Apple App Store in the India region. One of our team members interacted with the application, scanning five similar-looking pills — of which only two were recognised correctly. The choice of critiquing this application comes from its significance in addressing the challenges of medication identification in diverse environments, including airports.
In response, our proposed prototype addresses the specific challenge of ensuring safe medication compliance at airport security checkpoints. By deploying an AI-powered scanning device equipped with a database of prohibited medicines, our prototype seeks to assist airport officials in identifying and removing restricted items effectively — and takes the pressure of accurately labelling medicines away from travellers. To evaluate the efficacy and usability of our prototype, we adapted a heuristic approach and conducted user testing and speculative design sessions.
Smart Pill Identifier deploys AI methodologies with the aim of automating the detection process, minimising the effects of manual inspection and human error. The app's algorithm matches pills by parameters — inscription, colour, shape and size — and gives users detailed information about the medications, including description, side effects, and drug interactions.
At its core, the application uses computer vision, Natural Language Processing (NLP), and machine learning: the computer vision technology analyses images using colour, shape and inscriptions to facilitate accurate identification2, while the NLP technology enables the app to process and interpret prescription drug information34. Although the application has a promising concept, it poses several critical problems — ethical questions including privacy and security, algorithmic bias, accuracy, and reliability.
We envisioned a system where medicines are placed on a conveyor belt to be scanned and classified into one of three categories: Approved, Unapproved (needing documentation), or Illegal. A multi-class image classifier trained using Liner.ai was developed to identify and tag medicines — 31 classes in total, each with a minimum of 25 labelled images, with 5 classes flagged as illegal and 4 as unapproved.
We initially trained the model on images of medicines against a plain background, to simulate real-life operation of the scanner. However, while simulating the interface with web technologies, we found the model often misidentified human faces as medicines, reducing its real-life accuracy. To work around this, we added a "human" class with labelled images of all team members, and wrote logic to disregard anything tagged human.
The reported classification accuracy on Liner.ai was 97%. We deliberately used the classes with the highest image counts as the illegal classes, since the implications of misidentifying an illegal drug as legal are far higher than the reverse — a generated confusion matrix helped surface this risk, which we then corrected by increasing the image count for illegal classes specifically. We also found that duplicating images for training actually made classification performance slightly worse.
To keep the process fair, the model generates a report at the end listing approved, unapproved, and illegal medicines, handed to the passenger — who can then obtain equivalent medicines from a local pharmacy. The model is designed to improve over time, as officers can manually intervene and re-tag misidentified medicines, allowing for co-learning and adaptation following the principles of Human-Centered Machine Learning.
Our web interface is structured into distinct sections for traveller details, a scanner, action buttons, and categorised buckets for accepted and rejected items — a logical hierarchy that guides attention from left to right and minimises cognitive strain. Medicine entries are shown in two formats: a colour-coded list (green/red for acceptance/rejection) and a detailed table with thumbnails and rejection reasons, with filters and search to help officers quickly locate information. Two primary buttons — send report, and rescan — are consolidated into a single button stack to reduce complexity, and a bottom navigation bar switches between scanner view and settings, leaning on Fitts's law to prioritise accessibility.
Four participants were recruited for user testing to inform and validate the UI/UX design of the web application. The team adapted heuristic evaluation and a think-aloud technique to gather comprehensive feedback: users were introduced to the application, asked to observe the screens and share their understanding of its features, then asked to complete a task while the team observed for difficulties in navigation or confusion over features. The interviews were a mix of online and offline, semi-structured sessions, asking users to evaluate the interface and comment on the designed user flows.
The iterative nature of this heuristic evaluation let us focus on the essentials and brainstorm from a user's perspective — while also accounting for external factors, like the usage environment being fast-paced with a high influx of people, requiring processing in a short window. We also dropped some ideas from the first version of the application based on user feedback.
The overall experience of designing the application was rewarding — we evolved as a team throughout the coursework. Although there were initial challenges with Lobe.ai, we came away with a deeper understanding of the inner workings of a machine learning model. Real-time inferences using our initial model trained on Lobe appeared to have better accuracy than the model generated with Liner, using identical datasets — we aim to tweak Liner's training parameters to try to achieve a better real-life experience, since frequent misclassification can negatively impact officers' work and passengers' travel experiences, and could result in legal action.
Ethnographic studies to better understand usage scenarios, and to integrate this tool into existing baggage-scanning workflows, would improve the experience for all stakeholders — as would testing directly with airport security officials, to surface operational pain points.
Further thought needs to be given to identifying partly used strips, or medicines not stored in their original packaging. Workflows to keep models up to date as medicine packaging changes must be implemented, and clear signage explaining the process to passengers — including what to do if medicines are rejected or need additional documentation — should be part of the deployed system.