Quality filtering and deduplication pipeline

Summary
Pipeline for doing quality filtering and deduplication of data as a preparation for LLM training.
More information & hyperlinks
Web resources: https://ec.europa.eu/research/participants/documents/downloadPublic?documentIds=080166e51779844c&appId=PPGMS
Search
Search on Google
Search on Bing
Search on DuckDuckGo
Associated projects
Organisations
  • Not specified
Powered by Chris Decubber BV