Knowledge aggregation is the mode shift and innovation path of the development and utilization of literature resources in the era of digital wisdom. Through field research, it is found that the red literature resources of the Jiangsu and Zhejiang Military Region of the New Fourth Army show a multi-modal prosperity pattern of stratification and blocks, while there are several problems that restrict the realization of their overall memory and cultural value, such as the diversity of subjects, different standards, insufficient correlation and integration, insufficient online and offline interaction and so on. Therefore, combined with the resource structure of red literature dominated by "time-space-event or person", a multi-modal knowledge aggregation model of red literature resources based on semantic association is designed from four levels: resource collection and processing, ontology construction, association aggregation and knowledge mining. Taking general Su Yu and his deeds as an example to design metadata framework, ontology model and the GIS visualization of his life story, so as to carry out empirical research, which provides reference for the in-depth development of multi-modal red literature resources for knowledge discovery and service in the era of digital wisdom.