Non-Visual Information Transmission for the Blind and Visually Impaired

Tactile and multi-modal AI interfaces that make visual information (color, art, text) accessible.

This line of work, carried out at Sungkyunkwan University’s H-Lab, develops assistive technologies that translate visual information into non-visual channels for blind and visually impaired users.

Contributions include a reconfigurable tactile platform for non-visual information transmission and ColorWatch, a color-perceptual spatial tactile interface (Jabbar et al., 2021) (Lee et al., 2020). On the multi-modal AI side, the AI-Ekphrasis framework uses transformer-based foundation models to retrieve fine-grained text descriptions for images, improving mean retrieval ranking to 8.2 versus a 15.5 baseline (Jabbar et al., 2022). Related work on automatically generating art captions for visually impaired audiences received a Best Paper Award (Lee et al., 2021), and the tactile color-display device developed alongside this project was granted a Korean patent (Cho et al., 2020).

References

2022

  1. Electronics
    AI Ekphrasis: Multi-Modal Learning with Foundation Models for Fine-Grained Poetry Retrieval
    Muhammad Shahid Jabbar, Jitae Shin, and Jun-Dong Cho
    Electronics, 2022
    PDF

2021

  1. Electronics
    ColorWatch: Color Perceptual Spatial Tactile Interface for People with Visual Impairments
    Muhammad Shahid Jabbar, Chung-Heon Lee, and Jun-Dong Cho
    Electronics, 2021
    PDF
  2. SPIE
    Language of gleam: impressionism artwork automatic caption generation for people with visual impairments
    Dongmyeong Lee, Hyegyeong Hwang, Muhammad Shahid Jabbar, and 1 more author
    In Thirteenth International Conference on Machine Vision, 2021
    Best Paper Award Bib PDF

2020

  1. ICTC
    ColorWatch: Color-Tactile Interpretation to Improve Color Perception and Accessibility for People with Visual Impairments
    Chung-Heon Lee, Muhammad Shahid Jabbar, and Jun-Dong Cho
    In 2020 International Conference on Information and Communication Technology Convergence (ICTC), 2020
    PDF
  2. Patent
    Tactile device for displaying color information
    Jun-Dong Cho, Chung-Heon Lee, and Muhammad Shahid Jabbar
    2020
    Korean Patent Application No. 10-2020-0148650
    Bib