关键词： 清除重复, 数据清理, 数据核对, 筛选入库, 数据仓库, 数据导出

Abstract: When a database is established,judgment on data duplication is crucial for its administration,which will be difficult without accurate keywords for reference.The commonly used methods-Hash technology,fixed-sized partition detection technology,sliding block technology,content-defined chunking detection technology,and fingerprint data exploitation,require a large amount of processing time for the detection and removal of duplication.This paper describes the ASP mass data storage method and duplication screening method,and verifies the robustness and validity of these methods.It is shown that the heavy workload of database management for operators can be greatly reduced.

Key words: duplication removal, data cleaning, data check, screening and storage, data warehouse, data export