03.Windows编译运行VideoRAG算法

一、项目介绍1.1 仓库地址GIT URL: https://github.com/HKUDS/VideoRAG/tree/main/VideoRAG-algorithm1.2 项目说明VideoRAG 引入了一种新颖的双通道架构该架构协同结合图驱动的文本知识归一化用于建模跨视频语义关系以及层次化的多模态上下文编码以保留时空视觉模式通过动态构建的知识图实现无限制长度的视频理解这些知识图在多视频上下文中保持语义一致性同时通过自适应的多模态融合机制优化检索效率。高效的极长上下文视频处理利用单个 NVIDIA RTX 3090 GPU24G理解数百小时的视频内容结构化视频知识索引多模态知识索引框架将数百小时的视频提炼成简洁、结构化的知识图谱用于全面响应的多模态检索多模态检索范式将文本语义和视觉内容对齐以识别最相关的视频实现全面响应新建的长视频基准新建立的 LongerVideos 基准测试包含超过 160 个视频总时长超过 134 小时涵盖讲座、纪录片和娱乐内容二、配置运行环境2.1 conda环境conda create--namevideoragpython3.11conda activate videorag# Core numerical and deep learning librariespipinstallnumpy1.26.4 pipinstalltorch2.1.2torchvision0.16.2torchaudio2.1.2 pipinstallaccelerate0.30.1 pipinstallbitsandbytes0.43.1# Video processing utilitiespipinstallmoviepy1.0.3 pipinstallgithttps://github.com/facebookresearch/pytorchvideo.git28fe037d212663c6a24f373b94cc5d478c8c1a1d pipinstall--no-deps githttps://github.com/facebookresearch/ImageBind.git3fcf5c9039de97f6ff5528ee4a9dce903c5979b3# Multi-modal and vision librariespipinstalltimm ftfy regex einops fvcore eva-decord0.6.1 iopath matplotlib types-regex cartopy# Audio processing and vector databasespipinstallctranslate24.4.0faster_whisper1.0.3 neo4j hnswlib xxhash nano-vectordb# Language models and utilitiespipinstalltransformers4.37.1 pipinstalltiktoken openai tenacity pipinstallollama0.5.32.2 模型下载下载 MiniCPM-V、Whisper 和 ImageBind 所需的检查点到从仓库根目录。# Ensure git-lfs is installedgitlfsinstall# Download MiniCPM-V modelgitlfs clone https://huggingface.co/openbmb/MiniCPM-V-2_6-int4# Download Whisper modelgitlfs clone https://huggingface.co/Systran/faster-distil-whisper-large-v3# Download ImageBind checkpointmkdir.checkpointscd.checkpointswgethttps://dl.fbaipublicfiles.com/imagebind/imagebind_huge.pthcd../三、问题修复如果上面两个项目没有提前clone可以修改以下截图所圈的地方。modelWhisperModel(Systran/faster-distil-whisper-large-v3,devicecpu,compute_typeint8)四、使用方式目前VideoRAG 仅在英语环境中进行了测试。要处理多种语言的视频建议修改 asr.py 中的 WhisperModel 。更多详情请参考 faster-whisper。首先让 VideoRAG 从给定的视频中提取和索引知识只需一个配备 24GB 内存的 GPU 即可例如 RTX 3090importosimportloggingimportwarningsimportmultiprocessing warnings.filterwarnings(ignore)logging.getLogger(httpx).setLevel(logging.WARNING)# Please enter your openai keyos.environ[OPENAI_API_KEY]fromvideorag._llmimportopenai_4o_mini_configfromvideoragimportVideoRAG,QueryParamif__name____main__:multiprocessing.set_start_method(spawn)# Please enter your video file path in this list; there is no limit on the length.# Here is an example; you can use your own videos instead.video_paths[movies/Iron-Man.mp4,movies/Spider-Man.mkv,]videoragVideoRAG(llmopenai_4o_mini_config,working_dirf./videorag-workdir)videorag.insert_video(video_path_listvideo_paths)然后对视频提出任何问题以下是一个示例importosimportloggingimportwarningsimportmultiprocessing warnings.filterwarnings(ignore)logging.getLogger(httpx).setLevel(logging.WARNING)# Please enter your openai keyos.environ[OPENAI_API_KEY]fromvideorag._llmimport*fromvideoragimportVideoRAG,QueryParamif__name____main__:multiprocessing.set_start_method(spawn)queryWhat is the relationship between Iron Man and Spider-Man? How do they meet, and how does Iron Man help Spider-Man?paramQueryParam(modevideorag)# if param.wo_reference False, VideoRAG will add reference to video clips in the responseparam.wo_referenceTruevideoragvideoragVideoRAG(llmopenai_4o_mini_config,working_dirf./videorag-workdir)videorag.load_caption_model(debugFalse)responsevideorag.query(queryquery,paramparam)print(response)