History in the Making: A Case Study of AI Application
This piece was written by Hannah Storch.
“She gave Black people an opportunity to look at themselves on a big screen as something beautiful when all that was there before spoke to our degradation. In her we found another dimension to being black in our time.” – Harry Belafonte of Etta Moten Barnett
Etta Moten Barnett: A HistoryMaker
Etta Moten Barnett was an African American singer and actress known for her legendary roles in Sugar Hill (1931) and Porgy and Bess (1942) on Broadway as well as in the film Flying Down to Rio (1933) and in many other notable theatrical performances. Along with her husband Claude Barnett, founder of the Associated Negro Press, Etta Moten was involved in many philanthropic efforts in the United States and Africa and served as a pillar of her community.
The HistoryMakers is an organization that documents and promotes stories like those of Etta Moten, to create a more inclusive record of American history and has been recording African American oral histories since 1999. This one-of-a-kind collection is housed permanently at the Library of Congress and contains almost 3,400 video oral history interviews of African American leaders from 413 cities and towns, Mexico, the Caribbean, and Norway across a variety of disciplines – including the arts, business, civic engagement, education, entertainment, law, the media, medicine. With education as its mission, The HistoryMakers website and digital archive provide an unprecedented and irreplaceable record of African American lives, accomplishments, and contributions through unique first-person testimony.
With the help of Digital Transitions’ service division DT Heritage, The HistoryMakers are now leveraging digital images of items from the personal collections of their HistoryMakers, including from the collection of Etta Moten and Claude Barnett, to construct a more holistic digital record of their lives. At DT Heritage we have been able to use our integrated scanning processing pipeline and Artificial Intelligence (AI) application to assist in the telling of these influential stories by providing enhanced digital assets that promote accessibility, prompt research, and provide valuable visual context.
Photograph of Etta Moten Barnett courtesy of The HistoryMakers
Initial Pilot Project
The pilot project for this digitization program was from Etta Moten and Claude Barnett’s personal collection and consisted of several scrapbooks as well as photographic prints, correspondence, and newspapers and newspaper clippings relating to their lives. Personal collections, such as this, create a unique challenge for full-scale digitization and cataloguing efforts as there is often little to no existing metadata information describing the contents. What little information there is relies heavily on the personal knowledge of the individuals cataloguing the items. Given our extensive experience dealing with personal collections and the advice of the Digitization Committee for this mass-digitization project, DT Heritage took a “Digitize First, Process Second” approach, meaning that we were able to use rapid capture photography to image the materials first and then use AI to derive item-level descriptive metadata information from the images themselves. By using Digital Transitions’ scanning hardware and our RAW rapid capture workflow, we were able to preserve all of the information recorded by the system at the time of capture, not lose any information due to compression, and create high-quality digital images to integrate into our processing pipeline. This created the perfect platform for AI implementation and image analysis. We also used our proprietary Optical Character Recognition (OCR) software to create searchable transcription records of the correspondence and newspapers so that researchers and the general public could learn more about Etta Moten and her life through keyword and subject searches.
President Dwight D. Eisenhower and Claude Barnett as identified by PixelFlow image analysis.
Artificial Intelligence Application
Along with creating digital preservation-grade images of the collection, our DT PixelFlow software uses computer vision artificial intelligence to describe an item’s material, text, and image content through material analysis, text analysis, and image analysis respectively. For the Etta Moten Barnett Collection, image analysis was the most crucial of the three as there was little descriptive information or contextualization for the photographs that made up the majority of the collection. Using our AI capabilities, we provided object recognition, keyword extraction, and customized notable person detection. For this project, we applied our software to the collection in order to extract keywords and context from the photographic prints – as well as recognize Etta Moten, Claude Barnett, and other notable individuals from their life experiences throughout the collection. Unlike generic facial detection options on the market, our notable person detection cross references the approximate age of the individual, their date of birth and death, and the approximate date range of the physical collection, to confirm that a detected face is a plausible identification.
Since we understood the richness of Etta Moten’s history and how impactful she was within her community, we wanted to ensure that we were capturing as much information as possible from the digital images so we implemented two different types of person detection/face recognition. The initial process was Notable Individual Detection which compared the faces identified within the collection against an existing database of faces of notable people. This proved an excellent source of information for discovering individuals that we didn’t know were in the collection, such as President Dwight D. Eisenhower, American pianist and composer Hadda Brooks, and actor Laurence Olivier.
Knowing that the collection belonged to Etta Moten and Claude Barnett and that they were likely to be in many of the photographic prints, we also utilized our custom-trained facial recognition model. By using several previously-identified images of Etta and Claude at the beginning of the project, we were able to train our AI software to recognize their particular faces and the results were extraordinary. Even within class photos with many difficult-to-distinguish faces and people, our software was able to recognize Claude Barnett with astounding accuracy. In order to establish a criteria of confidence in the accuracy of the software’s ability to recognize faces, we also provided an accuracy percentage along with the descriptive metadata. This accuracy percentage served as a confidence rating for The HistoryMakers who could then take a manual look at any images that fell below a certain threshold to confirm the software’s findings.
Etta Moten and Claude Barnett identified by custom-trained AI notable person detection.
Claude Barnett identified among a crowd of people in a group photo by custom-trained notable person detection.
Identifications of Etta Moten and Claude Barnett through custom-trained AI facial recognition and notable person detection.
Along with discovering keywords and notable people within the collection and providing valuable context to Etta Moten Barnett’s story, our DT PixelFlow software was able to input this information into a workable spreadsheet. By working closely with The HistoryMakers and their website team at ThirdWave LLC., we were able to incorporate any descriptive metadata our software was able to find as well as technical metadata about the digital assets themselves into a Dublin Core spreadsheet that they could easily use with their online platform.
Significance
Since 1999, The HistoryMakers has amassed a collection of first-person oral history interviews with 3,000 influential African American ArtMakers, BusinessMakers, CivicMakers, EducationMakers, EntertainmentMakers, LawMakers, MediaMakes, MedicalMakers, MilitaryMakers, MusicMakers, PoliticalMakers, ReligionMakers, ScienceMakers, SportsMakers, and StyleMakers that is still growing. Through DT Heritage’s partnership with the organization, we look forward to adding valuable visual context to the voices of these powerful HistoryMakers and creating descriptive records during the digitization process for collections that previously had none. By using artificial intelligence to recognize people and contextualize moments within these HistoryMakers’ lives through object recognition and keyword extraction, we are able to enhance their narratives and work with The HistoryMakers to create an unprecedented repository of stories and digital assets and an invaluable resource of American history.
Do you have a need for Artificial Intelligence applications within
your own collection?