idanoo/autobrr
1
0
Fork 0
mirror of https://github.com/idanoo/autobrr synced 2025-07-23 00:39:13 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions

feat(filters): skip duplicates (#1711)

Browse source 
* feat(filters): skip duplicates

* fix: add interface instead of any

* fix(filters): tonullint

* feat(filters): skip dupes check month day

* chore: cleanup

* feat(db): set autoincrement id

* feat(filters): add repack and proper to dupe profile

* feat(filters): add default dupe profiles

* feat(duplicates): check audio and website

* feat(duplicates): update tests

* feat(duplicates): add toggles on addform

* feat(duplicates): fix sqlite upgrade path and initialize duplicate profiles

* feat(duplicates): simplify sqlite upgrade

avoiding temp table and unwieldy select.  Besides, FK constraints
are turned off anyway in #229.

* feat(duplicates): change CheckIsDuplicateRelease treatment of PROPER and REPACK

"Proper" and "Repack" are not parallel to the other conditions like "Title",
so they do not belong as dedup conditions.  "PROPER" means there was an issue in
the previous release, and so a PROPER is never a duplicate, even if it replaces
another PROPER.  Similarly, "REPACK" means there was an issue in the previous
release by that group, and so it is a duplicate only if we previously took a
release from a DIFFERENT group.

I have not removed Proper and Repack from the UI or the schema yet.

* feat(duplicates): update postgres schema to match sqlite

* feat(duplicates): fix web build errors

* feat(duplicates): fix postgres errors

* feat(filters): do leftjoin for duplicate profile

* fix(filters): partial update dupe profile

* go fmt `internal/domain/filter.go`

* feat(duplicates): restore straightforward logic for proper/repack

* feat(duplicates): remove mostly duplicate TV duplicate profiles

Having one profile seems the cleanest.  If somebody wants multiple
resolutions then they can add Resolution to the duplicate profile.
Tested this profile with both weekly episodic releases and daily
show releases.

* feat(release): add db indexes and sub_title

* feat(release): add IsDuplicate tests

* feat(release): update action handler

* feat(release): add more tests for skip duplicates

* feat(duplicates): check audio

* feat(duplicates): add more tests

* feat(duplicates): match edition cut and more

* fix(duplicates): tests

* fix(duplicates): missing imports

* fix(duplicates): tests

* feat(duplicates): handle sub_title edition and language in ui

* fix(duplicates): tests

* feat(duplicates): check name against normalized hash

* fix(duplicates): tests

* chore: update .gitignore to ignore .pnpm-store

* fix: tests

* fix(filters): tests

* fix: bad conflict merge

* fix: update release type in test

* fix: use vendored hot-toast

* fix: release_test.go

* fix: rss_test.go

* feat(duplicates): improve title hashing for unique check

* feat(duplicates): further improve title hashing for unique check with lang

* feat(duplicates): fix tests

* feat(duplicates): add macros IsDuplicate and DuplicateProfile ID and name

* feat(duplicates): add normalized hash match option

* fix: headlessui-state prop warning

* fix(duplicates): add missing year in daily ep normalize

* fix(duplicates): check rejections len

---------

Co-authored-by: ze0s <ze0s@riseup.net>
This commit is contained in:
kenstir 2024-12-25 16:33:46 -05:00 • committed by GitHub
parent d153ac44b8
commit 4009554d10
No known key found for this signature in database
GPG key ID: B5690EEEBB952194
49 changed files with 3792 additions and 743 deletions
Download patch file Download diff file Expand all files Collapse all files

2
internal/domain/action.go
View file

@ -11,7 +11,7 @@ import (
)
type ActionRepo interface {
Store(ctx context.Context, action Action) (*Action, error)
Store(ctx context.Context, action *Action) error
StoreFilterActions(ctx context.Context, filterID int64, actions []*Action) ([]*Action, error)
FindByFilterID(ctx context.Context, filterID int, active *bool, withClient bool) ([]*Action, error)
List(ctx context.Context) ([]Action, error)

315
internal/domain/filter.go
View file

@ -91,88 +91,90 @@ type FilterQueryParams struct {
}
type Filter struct {
ID int `json:"id"`
Name string `json:"name"`
Enabled bool `json:"enabled"`
CreatedAt time.Time `json:"created_at"`
UpdatedAt time.Time `json:"updated_at"`
MinSize string `json:"min_size,omitempty"`
MaxSize string `json:"max_size,omitempty"`
Delay int `json:"delay,omitempty"`
Priority int32 `json:"priority"`
MaxDownloads int `json:"max_downloads,omitempty"`
MaxDownloadsUnit FilterMaxDownloadsUnit `json:"max_downloads_unit,omitempty"`
MatchReleases string `json:"match_releases,omitempty"`
ExceptReleases string `json:"except_releases,omitempty"`
UseRegex bool `json:"use_regex,omitempty"`
MatchReleaseGroups string `json:"match_release_groups,omitempty"`
ExceptReleaseGroups string `json:"except_release_groups,omitempty"`
AnnounceTypes []string `json:"announce_types,omitempty"`
Scene bool `json:"scene,omitempty"`
Origins []string `json:"origins,omitempty"`
ExceptOrigins []string `json:"except_origins,omitempty"`
Bonus []string `json:"bonus,omitempty"`
Freeleech bool `json:"freeleech,omitempty"`
FreeleechPercent string `json:"freeleech_percent,omitempty"`
SmartEpisode bool `json:"smart_episode"`
Shows string `json:"shows,omitempty"`
Seasons string `json:"seasons,omitempty"`
Episodes string `json:"episodes,omitempty"`
Resolutions []string `json:"resolutions,omitempty"` // SD, 480i, 480p, 576p, 720p, 810p, 1080i, 1080p.
Codecs []string `json:"codecs,omitempty"` // XviD, DivX, x264, h.264 (or h264), mpeg2 (or mpeg-2), VC-1 (or VC1), WMV, Remux, h.264 Remux (or h264 Remux), VC-1 Remux (or VC1 Remux).
Sources []string `json:"sources,omitempty"` // DSR, PDTV, HDTV, HR.PDTV, HR.HDTV, DVDRip, DVDScr, BDr, BD5, BD9, BDRip, BRRip, DVDR, MDVDR, HDDVD, HDDVDRip, BluRay, WEB-DL, TVRip, CAM, R5, TELESYNC, TS, TELECINE, TC. TELESYNC and TS are synonyms (you don't need both). Same for TELECINE and TC
Containers []string `json:"containers,omitempty"`
MatchHDR []string `json:"match_hdr,omitempty"`
ExceptHDR []string `json:"except_hdr,omitempty"`
MatchOther []string `json:"match_other,omitempty"`
ExceptOther []string `json:"except_other,omitempty"`
Years string `json:"years,omitempty"`
Months string `json:"months,omitempty"`
Days string `json:"days,omitempty"`
Artists string `json:"artists,omitempty"`
Albums string `json:"albums,omitempty"`
MatchReleaseTypes []string `json:"match_release_types,omitempty"` // Album,Single,EP
ExceptReleaseTypes string `json:"except_release_types,omitempty"`
Formats []string `json:"formats,omitempty"` // MP3, FLAC, Ogg, AAC, AC3, DTS
Quality []string `json:"quality,omitempty"` // 192, 320, APS (VBR), V2 (VBR), V1 (VBR), APX (VBR), V0 (VBR), q8.x (VBR), Lossless, 24bit Lossless, Other
Media []string `json:"media,omitempty"` // CD, DVD, Vinyl, Soundboard, SACD, DAT, Cassette, WEB, Other
PerfectFlac bool `json:"perfect_flac,omitempty"`
Cue bool `json:"cue,omitempty"`
Log bool `json:"log,omitempty"`
LogScore int `json:"log_score,omitempty"`
MatchCategories string `json:"match_categories,omitempty"`
ExceptCategories string `json:"except_categories,omitempty"`
MatchUploaders string `json:"match_uploaders,omitempty"`
ExceptUploaders string `json:"except_uploaders,omitempty"`
MatchRecordLabels string `json:"match_record_labels,omitempty"`
ExceptRecordLabels string `json:"except_record_labels,omitempty"`
MatchLanguage []string `json:"match_language,omitempty"`
ExceptLanguage []string `json:"except_language,omitempty"`
Tags string `json:"tags,omitempty"`
ExceptTags string `json:"except_tags,omitempty"`
TagsAny string `json:"tags_any,omitempty"`
ExceptTagsAny string `json:"except_tags_any,omitempty"`
TagsMatchLogic string `json:"tags_match_logic,omitempty"`
ExceptTagsMatchLogic string `json:"except_tags_match_logic,omitempty"`
MatchReleaseTags string `json:"match_release_tags,omitempty"`
ExceptReleaseTags string `json:"except_release_tags,omitempty"`
UseRegexReleaseTags bool `json:"use_regex_release_tags,omitempty"`
MatchDescription string `json:"match_description,omitempty"`
ExceptDescription string `json:"except_description,omitempty"`
UseRegexDescription bool `json:"use_regex_description,omitempty"`
MinSeeders int `json:"min_seeders,omitempty"`
MaxSeeders int `json:"max_seeders,omitempty"`
MinLeechers int `json:"min_leechers,omitempty"`
MaxLeechers int `json:"max_leechers,omitempty"`
ActionsCount int `json:"actions_count"`
ActionsEnabledCount int `json:"actions_enabled_count"`
IsAutoUpdated bool `json:"is_auto_updated"`
Actions []*Action `json:"actions,omitempty"`
External []FilterExternal `json:"external,omitempty"`
Indexers []Indexer `json:"indexers"`
Downloads *FilterDownloads `json:"-"`
Rejections []string `json:"-"`
RejectReasons *RejectionReasons `json:"-"`
ID int `json:"id"`
Name string `json:"name"`
Enabled bool `json:"enabled"`
CreatedAt time.Time `json:"created_at"`
UpdatedAt time.Time `json:"updated_at"`
MinSize string `json:"min_size,omitempty"`
MaxSize string `json:"max_size,omitempty"`
Delay int `json:"delay,omitempty"`
Priority int32 `json:"priority"`
MaxDownloads int `json:"max_downloads,omitempty"`
MaxDownloadsUnit FilterMaxDownloadsUnit `json:"max_downloads_unit,omitempty"`
MatchReleases string `json:"match_releases,omitempty"`
ExceptReleases string `json:"except_releases,omitempty"`
UseRegex bool `json:"use_regex,omitempty"`
MatchReleaseGroups string `json:"match_release_groups,omitempty"`
ExceptReleaseGroups string `json:"except_release_groups,omitempty"`
AnnounceTypes []string `json:"announce_types,omitempty"`
Scene bool `json:"scene,omitempty"`
Origins []string `json:"origins,omitempty"`
ExceptOrigins []string `json:"except_origins,omitempty"`
Bonus []string `json:"bonus,omitempty"`
Freeleech bool `json:"freeleech,omitempty"`
FreeleechPercent string `json:"freeleech_percent,omitempty"`
SmartEpisode bool `json:"smart_episode"`
Shows string `json:"shows,omitempty"`
Seasons string `json:"seasons,omitempty"`
Episodes string `json:"episodes,omitempty"`
Resolutions []string `json:"resolutions,omitempty"` // SD, 480i, 480p, 576p, 720p, 810p, 1080i, 1080p.
Codecs []string `json:"codecs,omitempty"` // XviD, DivX, x264, h.264 (or h264), mpeg2 (or mpeg-2), VC-1 (or VC1), WMV, Remux, h.264 Remux (or h264 Remux), VC-1 Remux (or VC1 Remux).
Sources []string `json:"sources,omitempty"` // DSR, PDTV, HDTV, HR.PDTV, HR.HDTV, DVDRip, DVDScr, BDr, BD5, BD9, BDRip, BRRip, DVDR, MDVDR, HDDVD, HDDVDRip, BluRay, WEB-DL, TVRip, CAM, R5, TELESYNC, TS, TELECINE, TC. TELESYNC and TS are synonyms (you don't need both). Same for TELECINE and TC
Containers []string `json:"containers,omitempty"`
MatchHDR []string `json:"match_hdr,omitempty"`
ExceptHDR []string `json:"except_hdr,omitempty"`
MatchOther []string `json:"match_other,omitempty"`
ExceptOther []string `json:"except_other,omitempty"`
Years string `json:"years,omitempty"`
Months string `json:"months,omitempty"`
Days string `json:"days,omitempty"`
Artists string `json:"artists,omitempty"`
Albums string `json:"albums,omitempty"`
MatchReleaseTypes []string `json:"match_release_types,omitempty"` // Album,Single,EP
ExceptReleaseTypes string `json:"except_release_types,omitempty"`
Formats []string `json:"formats,omitempty"` // MP3, FLAC, Ogg, AAC, AC3, DTS
Quality []string `json:"quality,omitempty"` // 192, 320, APS (VBR), V2 (VBR), V1 (VBR), APX (VBR), V0 (VBR), q8.x (VBR), Lossless, 24bit Lossless, Other
Media []string `json:"media,omitempty"` // CD, DVD, Vinyl, Soundboard, SACD, DAT, Cassette, WEB, Other
PerfectFlac bool `json:"perfect_flac,omitempty"`
Cue bool `json:"cue,omitempty"`
Log bool `json:"log,omitempty"`
LogScore int `json:"log_score,omitempty"`
MatchCategories string `json:"match_categories,omitempty"`
ExceptCategories string `json:"except_categories,omitempty"`
MatchUploaders string `json:"match_uploaders,omitempty"`
ExceptUploaders string `json:"except_uploaders,omitempty"`
MatchRecordLabels string `json:"match_record_labels,omitempty"`
ExceptRecordLabels string `json:"except_record_labels,omitempty"`
MatchLanguage []string `json:"match_language,omitempty"`
ExceptLanguage []string `json:"except_language,omitempty"`
Tags string `json:"tags,omitempty"`
ExceptTags string `json:"except_tags,omitempty"`
TagsAny string `json:"tags_any,omitempty"`
ExceptTagsAny string `json:"except_tags_any,omitempty"`
TagsMatchLogic string `json:"tags_match_logic,omitempty"`
ExceptTagsMatchLogic string `json:"except_tags_match_logic,omitempty"`
MatchReleaseTags string `json:"match_release_tags,omitempty"`
ExceptReleaseTags string `json:"except_release_tags,omitempty"`
UseRegexReleaseTags bool `json:"use_regex_release_tags,omitempty"`
MatchDescription string `json:"match_description,omitempty"`
ExceptDescription string `json:"except_description,omitempty"`
UseRegexDescription bool `json:"use_regex_description,omitempty"`
MinSeeders int `json:"min_seeders,omitempty"`
MaxSeeders int `json:"max_seeders,omitempty"`
MinLeechers int `json:"min_leechers,omitempty"`
MaxLeechers int `json:"max_leechers,omitempty"`
ActionsCount int `json:"actions_count"`
ActionsEnabledCount int `json:"actions_enabled_count"`
IsAutoUpdated bool `json:"is_auto_updated"`
Actions []*Action `json:"actions,omitempty"`
External []FilterExternal `json:"external,omitempty"`
Indexers []Indexer `json:"indexers"`
ReleaseProfileDuplicateID int64 `json:"release_profile_duplicate_id,omitempty"`
DuplicateHandling *DuplicateReleaseProfile `json:"release_profile_duplicate"`
Downloads *FilterDownloads `json:"-"`
Rejections []string `json:"-"`
RejectReasons *RejectionReasons `json:"-"`
}
type FilterExternal struct {
@ -219,80 +221,81 @@ const (
)
type FilterUpdate struct {
ID int `json:"id"`
Name *string `json:"name,omitempty"`
Enabled *bool `json:"enabled,omitempty"`
MinSize *string `json:"min_size,omitempty"`
MaxSize *string `json:"max_size,omitempty"`
Delay *int `json:"delay,omitempty"`
Priority *int32 `json:"priority,omitempty"`
AnnounceTypes *[]string `json:"announce_types,omitempty"`
MaxDownloads *int `json:"max_downloads,omitempty"`
MaxDownloadsUnit *FilterMaxDownloadsUnit `json:"max_downloads_unit,omitempty"`
MatchReleases *string `json:"match_releases,omitempty"`
ExceptReleases *string `json:"except_releases,omitempty"`
UseRegex *bool `json:"use_regex,omitempty"`
MatchReleaseGroups *string `json:"match_release_groups,omitempty"`
ExceptReleaseGroups *string `json:"except_release_groups,omitempty"`
MatchReleaseTags *string `json:"match_release_tags,omitempty"`
ExceptReleaseTags *string `json:"except_release_tags,omitempty"`
UseRegexReleaseTags *bool `json:"use_regex_release_tags,omitempty"`
MatchDescription *string `json:"match_description,omitempty"`
ExceptDescription *string `json:"except_description,omitempty"`
UseRegexDescription *bool `json:"use_regex_description,omitempty"`
Scene *bool `json:"scene,omitempty"`
Origins *[]string `json:"origins,omitempty"`
ExceptOrigins *[]string `json:"except_origins,omitempty"`
Bonus *[]string `json:"bonus,omitempty"`
Freeleech *bool `json:"freeleech,omitempty"`
FreeleechPercent *string `json:"freeleech_percent,omitempty"`
SmartEpisode *bool `json:"smart_episode,omitempty"`
Shows *string `json:"shows,omitempty"`
Seasons *string `json:"seasons,omitempty"`
Episodes *string `json:"episodes,omitempty"`
Resolutions *[]string `json:"resolutions,omitempty"` // SD, 480i, 480p, 576p, 720p, 810p, 1080i, 1080p.
Codecs *[]string `json:"codecs,omitempty"` // XviD, DivX, x264, h.264 (or h264), mpeg2 (or mpeg-2), VC-1 (or VC1), WMV, Remux, h.264 Remux (or h264 Remux), VC-1 Remux (or VC1 Remux).
Sources *[]string `json:"sources,omitempty"` // DSR, PDTV, HDTV, HR.PDTV, HR.HDTV, DVDRip, DVDScr, BDr, BD5, BD9, BDRip, BRRip, DVDR, MDVDR, HDDVD, HDDVDRip, BluRay, WEB-DL, TVRip, CAM, R5, TELESYNC, TS, TELECINE, TC. TELESYNC and TS are synonyms (you don't need both). Same for TELECINE and TC
Containers *[]string `json:"containers,omitempty"`
MatchHDR *[]string `json:"match_hdr,omitempty"`
ExceptHDR *[]string `json:"except_hdr,omitempty"`
MatchOther *[]string `json:"match_other,omitempty"`
ExceptOther *[]string `json:"except_other,omitempty"`
Years *string `json:"years,omitempty"`
Months *string `json:"months,omitempty"`
Days *string `json:"days,omitempty"`
Artists *string `json:"artists,omitempty"`
Albums *string `json:"albums,omitempty"`
MatchReleaseTypes *[]string `json:"match_release_types,omitempty"` // Album,Single,EP
ExceptReleaseTypes *string `json:"except_release_types,omitempty"`
Formats *[]string `json:"formats,omitempty"` // MP3, FLAC, Ogg, AAC, AC3, DTS
Quality *[]string `json:"quality,omitempty"` // 192, 320, APS (VBR), V2 (VBR), V1 (VBR), APX (VBR), V0 (VBR), q8.x (VBR), Lossless, 24bit Lossless, Other
Media *[]string `json:"media,omitempty"` // CD, DVD, Vinyl, Soundboard, SACD, DAT, Cassette, WEB, Other
PerfectFlac *bool `json:"perfect_flac,omitempty"`
Cue *bool `json:"cue,omitempty"`
Log *bool `json:"log,omitempty"`
LogScore *int `json:"log_score,omitempty"`
MatchCategories *string `json:"match_categories,omitempty"`
ExceptCategories *string `json:"except_categories,omitempty"`
MatchUploaders *string `json:"match_uploaders,omitempty"`
ExceptUploaders *string `json:"except_uploaders,omitempty"`
MatchRecordLabels *string `json:"match_record_labels,omitempty"`
ExceptRecordLabels *string `json:"except_record_labels,omitempty"`
MatchLanguage *[]string `json:"match_language,omitempty"`
ExceptLanguage *[]string `json:"except_language,omitempty"`
Tags *string `json:"tags,omitempty"`
ExceptTags *string `json:"except_tags,omitempty"`
TagsAny *string `json:"tags_any,omitempty"`
ExceptTagsAny *string `json:"except_tags_any,omitempty"`
TagsMatchLogic *string `json:"tags_match_logic,omitempty"`
ExceptTagsMatchLogic *string `json:"except_tags_match_logic,omitempty"`
MinSeeders *int `json:"min_seeders,omitempty"`
MaxSeeders *int `json:"max_seeders,omitempty"`
MinLeechers *int `json:"min_leechers,omitempty"`
MaxLeechers *int `json:"max_leechers,omitempty"`
Actions []*Action `json:"actions,omitempty"`
External []FilterExternal `json:"external,omitempty"`
Indexers []Indexer `json:"indexers,omitempty"`
ID int `json:"id"`
Name *string `json:"name,omitempty"`
Enabled *bool `json:"enabled,omitempty"`
MinSize *string `json:"min_size,omitempty"`
MaxSize *string `json:"max_size,omitempty"`
Delay *int `json:"delay,omitempty"`
Priority *int32 `json:"priority,omitempty"`
AnnounceTypes *[]string `json:"announce_types,omitempty"`
MaxDownloads *int `json:"max_downloads,omitempty"`
MaxDownloadsUnit *FilterMaxDownloadsUnit `json:"max_downloads_unit,omitempty"`
MatchReleases *string `json:"match_releases,omitempty"`
ExceptReleases *string `json:"except_releases,omitempty"`
UseRegex *bool `json:"use_regex,omitempty"`
MatchReleaseGroups *string `json:"match_release_groups,omitempty"`
ExceptReleaseGroups *string `json:"except_release_groups,omitempty"`
MatchReleaseTags *string `json:"match_release_tags,omitempty"`
ExceptReleaseTags *string `json:"except_release_tags,omitempty"`
UseRegexReleaseTags *bool `json:"use_regex_release_tags,omitempty"`
MatchDescription *string `json:"match_description,omitempty"`
ExceptDescription *string `json:"except_description,omitempty"`
UseRegexDescription *bool `json:"use_regex_description,omitempty"`
Scene *bool `json:"scene,omitempty"`
Origins *[]string `json:"origins,omitempty"`
ExceptOrigins *[]string `json:"except_origins,omitempty"`
Bonus *[]string `json:"bonus,omitempty"`
Freeleech *bool `json:"freeleech,omitempty"`
FreeleechPercent *string `json:"freeleech_percent,omitempty"`
SmartEpisode *bool `json:"smart_episode,omitempty"`
Shows *string `json:"shows,omitempty"`
Seasons *string `json:"seasons,omitempty"`
Episodes *string `json:"episodes,omitempty"`
Resolutions *[]string `json:"resolutions,omitempty"` // SD, 480i, 480p, 576p, 720p, 810p, 1080i, 1080p.
Codecs *[]string `json:"codecs,omitempty"` // XviD, DivX, x264, h.264 (or h264), mpeg2 (or mpeg-2), VC-1 (or VC1), WMV, Remux, h.264 Remux (or h264 Remux), VC-1 Remux (or VC1 Remux).
Sources *[]string `json:"sources,omitempty"` // DSR, PDTV, HDTV, HR.PDTV, HR.HDTV, DVDRip, DVDScr, BDr, BD5, BD9, BDRip, BRRip, DVDR, MDVDR, HDDVD, HDDVDRip, BluRay, WEB-DL, TVRip, CAM, R5, TELESYNC, TS, TELECINE, TC. TELESYNC and TS are synonyms (you don't need both). Same for TELECINE and TC
Containers *[]string `json:"containers,omitempty"`
MatchHDR *[]string `json:"match_hdr,omitempty"`
ExceptHDR *[]string `json:"except_hdr,omitempty"`
MatchOther *[]string `json:"match_other,omitempty"`
ExceptOther *[]string `json:"except_other,omitempty"`
Years *string `json:"years,omitempty"`
Months *string `json:"months,omitempty"`
Days *string `json:"days,omitempty"`
Artists *string `json:"artists,omitempty"`
Albums *string `json:"albums,omitempty"`
MatchReleaseTypes *[]string `json:"match_release_types,omitempty"` // Album,Single,EP
ExceptReleaseTypes *string `json:"except_release_types,omitempty"`
Formats *[]string `json:"formats,omitempty"` // MP3, FLAC, Ogg, AAC, AC3, DTS
Quality *[]string `json:"quality,omitempty"` // 192, 320, APS (VBR), V2 (VBR), V1 (VBR), APX (VBR), V0 (VBR), q8.x (VBR), Lossless, 24bit Lossless, Other
Media *[]string `json:"media,omitempty"` // CD, DVD, Vinyl, Soundboard, SACD, DAT, Cassette, WEB, Other
PerfectFlac *bool `json:"perfect_flac,omitempty"`
Cue *bool `json:"cue,omitempty"`
Log *bool `json:"log,omitempty"`
LogScore *int `json:"log_score,omitempty"`
MatchCategories *string `json:"match_categories,omitempty"`
ExceptCategories *string `json:"except_categories,omitempty"`
MatchUploaders *string `json:"match_uploaders,omitempty"`
ExceptUploaders *string `json:"except_uploaders,omitempty"`
MatchRecordLabels *string `json:"match_record_labels,omitempty"`
ExceptRecordLabels *string `json:"except_record_labels,omitempty"`
MatchLanguage *[]string `json:"match_language,omitempty"`
ExceptLanguage *[]string `json:"except_language,omitempty"`
Tags *string `json:"tags,omitempty"`
ExceptTags *string `json:"except_tags,omitempty"`
TagsAny *string `json:"tags_any,omitempty"`
ExceptTagsAny *string `json:"except_tags_any,omitempty"`
TagsMatchLogic *string `json:"tags_match_logic,omitempty"`
ExceptTagsMatchLogic *string `json:"except_tags_match_logic,omitempty"`
MinSeeders *int `json:"min_seeders,omitempty"`
MaxSeeders *int `json:"max_seeders,omitempty"`
MinLeechers *int `json:"min_leechers,omitempty"`
MaxLeechers *int `json:"max_leechers,omitempty"`
ReleaseProfileDuplicateID *int64 `json:"release_profile_duplicate_id,omitempty"`
Actions []*Action `json:"actions,omitempty"`
External []FilterExternal `json:"external,omitempty"`
Indexers []Indexer `json:"indexers,omitempty"`
}
func (f *Filter) Validate() error {

8
internal/domain/macros.go
View file

@ -50,6 +50,7 @@ type Macro struct {
IndexerIdentifierExternal string
IndexerName string
InfoUrl string
IsDuplicate bool
Language []string
Leechers int
LogScore int
@ -66,6 +67,8 @@ type Macro struct {
Seeders int
Size uint64
SizeString string
SkipDuplicateProfileID int64
SkipDuplicateProfileName string
Source string
Tags string
Title string
@ -123,6 +126,7 @@ func NewMacro(release Release) Macro {
IndexerIdentifierExternal: release.Indexer.IdentifierExternal,
IndexerName: release.Indexer.Name,
InfoUrl: release.InfoURL,
IsDuplicate: release.IsDuplicate,
Language: release.Language,
Leechers: release.Leechers,
LogScore: release.LogScore,
@ -140,6 +144,8 @@ func NewMacro(release Release) Macro {
Size: release.Size,
SizeString: humanize.Bytes(release.Size),
Source: release.Source,
SkipDuplicateProfileID: release.SkipDuplicateProfileID,
SkipDuplicateProfileName: release.SkipDuplicateProfileName,
Tags: strings.Join(release.Tags, ", "),
Title: release.Title,
TorrentDataRawBytes: release.TorrentDataRawBytes,
@ -149,7 +155,7 @@ func NewMacro(release Release) Macro {
TorrentPathName: release.TorrentTmpFile,
TorrentUrl: release.DownloadURL,
TorrentTmpFile: release.TorrentTmpFile,
Type: release.Type,
Type: release.Type.String(),
Uploader: release.Uploader,
RecordLabel: release.RecordLabel,
Website: release.Website,

3
internal/domain/macros_test.go
View file

@ -5,6 +5,7 @@ package domain
import (
"fmt"
"github.com/moistari/rls"
"testing"
"time"
@ -231,7 +232,7 @@ func TestMacros_Parse(t *testing.T) {
{
name: "test_type",
release: Release{
Type: "episode",
Type: rls.Episode,
},
args: args{text: "Type: {{ .Type }}"},
want: "Type: episode",

277
internal/domain/release.go
View file

@ -7,7 +7,11 @@ import (
"bufio"
"bytes"
"context"
"crypto/md5"
"encoding/hex"
"fmt"
"golang.org/x/text/transform"
"golang.org/x/text/unicode/norm"
"html"
"io"
"math"
@ -18,6 +22,7 @@ import (
"strconv"
"strings"
"time"
"unicode"
"github.com/autobrr/autobrr/pkg/errors"
"github.com/autobrr/autobrr/pkg/sharedhttp"
@ -43,6 +48,11 @@ type ReleaseRepo interface {
GetActionStatus(ctx context.Context, req *GetReleaseActionStatusRequest) (*ReleaseActionStatus, error)
StoreReleaseActionStatus(ctx context.Context, status *ReleaseActionStatus) error
StoreDuplicateProfile(ctx context.Context, profile *DuplicateReleaseProfile) error
FindDuplicateReleaseProfiles(ctx context.Context) ([]*DuplicateReleaseProfile, error)
DeleteReleaseProfileDuplicate(ctx context.Context, id int64) error
CheckIsDuplicateRelease(ctx context.Context, profile *DuplicateReleaseProfile, release *Release) (bool, error)
}
type Release struct {
@ -55,6 +65,7 @@ type Release struct {
Implementation ReleaseImplementation `json:"implementation"` // irc, rss, api
Timestamp time.Time `json:"timestamp"`
AnnounceType AnnounceType `json:"announce_type"`
Type rls.Type `json:"type"` // rls.Type
InfoURL string `json:"info_url"`
DownloadURL string `json:"download_url"`
MagnetURI string `json:"-"`
@ -63,9 +74,11 @@ type Release struct {
TorrentTmpFile string `json:"-"`
TorrentDataRawBytes []byte `json:"-"`
TorrentHash string `json:"-"`
TorrentName string `json:"name"` // full release name
TorrentName string `json:"name"` // full release name
NormalizedHash string `json:"normalized_hash"` // normalized torrent name and md5 hashed
Size uint64 `json:"size"`
Title string `json:"title"` // Parsed title
Title string `json:"title"` // Parsed title
SubTitle string `json:"sub_title"` // Parsed secondary title for shows e.g. episode name
Description string `json:"-"`
Category string `json:"category"`
Categories []string `json:"categories,omitempty"`
@ -89,8 +102,11 @@ type Release struct {
Proper bool `json:"proper"`
Repack bool `json:"repack"`
Website string `json:"website"`
Hybrid bool `json:"hybrid"`
Edition []string `json:"edition"`
Cut []string `json:"cut"`
MediaProcessing string `json:"media_processing"` // Remux, Encode, Untouched
Artists string `json:"-"`
Type string `json:"type"` // Album,Single,EP
LogScore int `json:"-"`
HasCue bool `json:"-"`
HasLog bool `json:"-"`
@ -110,15 +126,183 @@ type Release struct {
AdditionalSizeCheckRequired bool `json:"-"`
AdditionalUploaderCheckRequired bool `json:"-"`
AdditionalRecordLabelCheckRequired bool `json:"-"`
IsDuplicate bool `json:"-"`
SkipDuplicateProfileID int64 `json:"-"`
SkipDuplicateProfileName string `json:"-"`
FilterID int `json:"-"`
Filter *Filter `json:"-"`
ActionStatus []ReleaseActionStatus `json:"action_status"`
}
// Hash return md5 hashed normalized release name
func (r *Release) Hash() string {
formatted := r.TorrentName
// for tv and movies we create the formatted title to have the best chance of matching
if r.IsTypeVideo() {
formatted = r.NormalizedTitle()
}
normalized := MustNormalize(formatted)
h := md5.Sum([]byte(normalized))
str := hex.EncodeToString(h[:])
return str
}
// MustNormalize applies the Normalize transform to s, returning a lower cased,
// clean form of s useful for matching titles.
func MustNormalize(s string) string {
s, _, err := transform.String(NewNormalizer(), s)
if err != nil {
panic(err)
}
return s
}
// NewNormalizer is a custom rls.Normalizer that keeps plus sign + for HDR10+ fx
// It creates a new a text transformer chain (similiar to
// NewCleaner) that normalizes text to lower case clean form useful for
// matching titles.
//
// See: https://go.dev/blog/normalization
func NewNormalizer() transform.Transformer {
return transform.Chain(
norm.NFD,
rls.NewCollapser(
true, true,
"`"+`':;~!@#%^*=()[]{}<>/?|\",`, " \t\r\n\f._",
func(r, prev, next rune) rune {
switch {
case r == '-' && unicode.IsSpace(prev):
return -1
case r == '$' && (unicode.IsLetter(prev) || unicode.IsLetter(next)):
return 'S'
case r == '£' && (unicode.IsLetter(prev) || unicode.IsLetter(next)):
return 'L'
case r == '$', r == '£':
return -1
}
return r
},
),
norm.NFC,
)
}
func (r *Release) NormalizedTitle() string {
var v []string
v = append(v, r.Title)
if r.Year > 0 && r.Month > 0 && r.Day > 0 {
v = append(v, fmt.Sprintf("%d %d %d", r.Year, r.Month, r.Day))
} else if r.Year > 0 {
v = append(v, fmt.Sprintf("%d", r.Year))
}
if len(r.Language) > 0 {
v = append(v, strings.Join(r.Language, " "))
}
if len(r.Cut) > 0 {
v = append(v, strings.Join(r.Cut, " "))
}
if len(r.Edition) > 0 {
v = append(v, strings.Join(r.Edition, " "))
}
if r.Season > 0 && r.Episode > 0 {
v = append(v, fmt.Sprintf("S%dE%d", r.Season, r.Episode))
} else if r.Season > 0 && r.Episode == 0 {
v = append(v, fmt.Sprintf("S%d", r.Season))
}
if r.Proper {
v = append(v, "PROPER")
}
if r.Repack {
v = append(v, r.RepackStr())
}
if r.Hybrid {
v = append(v, "HYBRiD")
}
if r.SubTitle != "" {
v = append(v, r.SubTitle)
}
if r.Resolution != "" {
v = append(v, r.Resolution)
}
if r.Website != "" {
v = append(v, r.Website)
}
if r.Region != "" {
v = append(v, r.Region)
}
if r.Source != "" {
v = append(v, r.Source)
}
// remux
if r.MediaProcessing == "REMUX" {
v = append(v, "REMUX")
}
if len(r.Codec) > 0 {
v = append(v, strings.Join(r.Codec, " "))
}
if len(r.HDR) > 0 {
v = append(v, strings.Join(r.HDR, " "))
}
if len(r.Audio) > 0 {
v = append(v, r.AudioString())
}
str := strings.Join(v, " ")
if r.Group != "" {
str = fmt.Sprintf("%s-%s", str, r.Group)
}
return str
}
func (r *Release) RepackStr() string {
if r.Other != nil {
if slices.Contains(r.Other, "REPACK") {
return "REPACK"
} else if slices.Contains(r.Other, "REREPACK") {
return "REREPACK"
} else if slices.Contains(r.Other, "REPACK2") {
return "REPACK2"
} else if slices.Contains(r.Other, "REPACK3") {
return "REPACK3"
}
}
return ""
}
func (r *Release) Raw(s string) rls.Release {
return rls.ParseString(s)
}
func (r *Release) ParseType(s string) {
r.Type = rls.ParseType(s)
}
func (r *Release) IsTypeVideo() bool {
return r.Type.Is(rls.Movie, rls.Series, rls.Episode)
}
type AnnounceType string
const (
@ -361,6 +545,10 @@ func NewRelease(indexer IndexerMinimal) *Release {
Implementation: ReleaseImplementationIRC,
Timestamp: time.Now(),
Tags: []string{},
Language: []string{},
Edition: []string{},
Cut: []string{},
Other: []string{},
Size: 0,
AnnounceType: AnnounceTypeNew,
}
@ -371,28 +559,42 @@ func NewRelease(indexer IndexerMinimal) *Release {
func (r *Release) ParseString(title string) {
rel := rls.ParseString(title)
r.Type = rel.Type.String()
r.Type = rel.Type
r.TorrentName = title
r.Source = rel.Source
r.Resolution = rel.Resolution
r.Region = rel.Region
if rel.Language != nil {
r.Language = rel.Language
}
r.Audio = rel.Audio
r.AudioChannels = rel.Channels
r.Codec = rel.Codec
r.Container = rel.Container
r.HDR = rel.HDR
r.Artists = rel.Artist
r.Language = rel.Language
r.Other = rel.Other
if rel.Other != nil {
r.Other = rel.Other
}
r.Proper = slices.Contains(r.Other, "PROPER")
r.Repack = slices.Contains(r.Other, "REPACK")
r.Repack = slices.Contains(r.Other, "REPACK") || slices.Contains(r.Other, "REREPACK")
r.Hybrid = slices.Contains(r.Other, "HYBRiD")
// TODO default to Encode and set Untouched for discs
if slices.Contains(r.Other, "REMUX") {
r.MediaProcessing = "REMUX"
}
if r.Title == "" {
r.Title = rel.Title
}
r.SubTitle = rel.Subtitle
if r.Season == 0 {
r.Season = rel.Series
@ -415,8 +617,22 @@ func (r *Release) ParseString(title string) {
r.Group = rel.Group
}
if r.Website == "" {
r.Website = rel.Collection
}
if rel.Cut != nil {
r.Cut = rel.Cut
}
if rel.Edition != nil {
r.Edition = rel.Edition
}
r.ParseReleaseTagsString(r.ReleaseTags)
r.extraParseSource(rel)
r.NormalizedHash = r.Hash()
}
func (r *Release) extraParseSource(rel rls.Release) {
@ -451,7 +667,7 @@ func (r *Release) extraParseSource(rel rls.Release) {
}
// check res to be 1080p or 2160p and codec to be AVC, HEVC or if other contains Remux, then set source to BluRay if it differs
if !basicContainsSlice(r.Source, []string{"WEB-DL", "BluRay", "UHD.BluRay"}) && basicContainsSlice(r.Resolution, []string{"1080p", "2160p"}) && basicContainsMatch(r.Codec, []string{"AVC", "HEVC"}) && basicContainsMatch(r.Other, []string{"REMUX"}) {
if !basicContainsSlice(r.Source, []string{"WEB-DL", "BluRay", "UHD.BluRay"}) && basicContainsSlice(r.Resolution, []string{"1080p", "2160p"}) && basicContainsMatch(r.Codec, []string{"AVC", "H.264", "H.265", "HEVC"}) && basicContainsMatch(r.Other, []string{"REMUX"}) {
// handle missing or unexpected source for some bluray releases
if r.Resolution == "1080p" {
r.Source = "BluRay"
@ -463,6 +679,10 @@ func (r *Release) extraParseSource(rel rls.Release) {
}
func (r *Release) ParseReleaseTagsString(tags string) {
if tags == "" {
return
}
cleanTags := CleanReleaseTags(tags)
t := ParseReleaseTagString(cleanTags)
@ -543,6 +763,20 @@ func (r *Release) OpenTorrentFile() error {
return nil
}
// AudioString takes r.Audio and r.AudioChannels and returns a string like "DDP Atmos 5.1"
func (r *Release) AudioString() string {
var audio []string
audio = append(audio, r.Audio...)
audio = append(audio, r.AudioChannels)
if len(audio) > 0 {
return strings.Join(audio, " ")
}
return ""
}
func (r *Release) DownloadTorrentFileCtx(ctx context.Context) error {
return r.downloadTorrentFile(ctx)
}
@ -992,3 +1226,30 @@ func getUniqueTags(target []string, source []string) []string {
return target
}
type DuplicateReleaseProfile struct {
ID int64 `json:"id"`
Name string `json:"name"`
Protocol bool `json:"protocol"`
ReleaseName bool `json:"release_name"`
Hash bool `json:"hash"`
Title bool `json:"title"`
SubTitle bool `json:"sub_title"`
Year bool `json:"year"`
Month bool `json:"month"`
Day bool `json:"day"`
Source bool `json:"source"`
Resolution bool `json:"resolution"`
Codec bool `json:"codec"`
Container bool `json:"container"`
DynamicRange bool `json:"dynamic_range"`
Audio bool `json:"audio"`
Group bool `json:"group"`
Season bool `json:"season"`
Episode bool `json:"episode"`
Website bool `json:"website"`
Proper bool `json:"proper"`
Repack bool `json:"repack"`
Edition bool `json:"edition"`
Language bool `json:"language"`
}

3
internal/domain/release_download_test.go
View file

@ -15,6 +15,7 @@ import (
"testing"
"time"
"github.com/moistari/rls"
"github.com/rs/zerolog"
)
@ -127,7 +128,7 @@ func TestRelease_DownloadTorrentFile(t *testing.T) {
Repack bool
Website string
Artists string
Type string
Type rls.Type
LogScore int
Origin string
Tags []string

713
internal/domain/release_test.go
View file

@ -6,6 +6,7 @@ package domain
import (
"testing"
"github.com/moistari/rls"
"github.com/stretchr/testify/assert"
)
@ -21,19 +22,20 @@ func TestRelease_Parse(t *testing.T) {
TorrentName: "Servant S01 2160p ATVP WEB-DL DDP 5.1 Atmos DV HEVC-FLUX",
},
want: Release{
TorrentName: "Servant S01 2160p ATVP WEB-DL DDP 5.1 Atmos DV HEVC-FLUX",
Title: "Servant",
Season: 1,
Episode: 0,
Resolution: "2160p",
Source: "WEB-DL",
Codec: []string{"HEVC"},
Audio: []string{"DDP", "Atmos"},
AudioChannels: "5.1",
HDR: []string{"DV"},
Group: "FLUX",
//Website: "ATVP",
Type: "series",
TorrentName: "Servant S01 2160p ATVP WEB-DL DDP 5.1 Atmos DV HEVC-FLUX",
NormalizedHash: "9b73e77d51fb0b69367ea96c761577b0",
Title: "Servant",
Season: 1,
Episode: 0,
Resolution: "2160p",
Source: "WEB-DL",
Codec: []string{"HEVC"},
Audio: []string{"DDP", "Atmos"},
AudioChannels: "5.1",
HDR: []string{"DV"},
Group: "FLUX",
Website: "ATVP",
Type: rls.Series,
},
},
{
@ -42,18 +44,20 @@ func TestRelease_Parse(t *testing.T) {
TorrentName: "Servant.S01.2160p.ATVP.WEB-DL.DDP.5.1.Atmos.DV.HEVC-FLUX",
},
want: Release{
TorrentName: "Servant.S01.2160p.ATVP.WEB-DL.DDP.5.1.Atmos.DV.HEVC-FLUX",
Title: "Servant",
Season: 1,
Episode: 0,
Resolution: "2160p",
Source: "WEB-DL",
Codec: []string{"HEVC"},
Audio: []string{"DDP", "Atmos"},
AudioChannels: "5.1",
HDR: []string{"DV"},
Group: "FLUX",
Type: "series",
TorrentName: "Servant.S01.2160p.ATVP.WEB-DL.DDP.5.1.Atmos.DV.HEVC-FLUX",
NormalizedHash: "9b73e77d51fb0b69367ea96c761577b0",
Title: "Servant",
Season: 1,
Episode: 0,
Resolution: "2160p",
Source: "WEB-DL",
Codec: []string{"HEVC"},
Audio: []string{"DDP", "Atmos"},
AudioChannels: "5.1",
HDR: []string{"DV"},
Group: "FLUX",
Website: "ATVP",
Type: rls.Series,
},
},
{
@ -63,20 +67,22 @@ func TestRelease_Parse(t *testing.T) {
ReleaseTags: "MKV / 2160p / WEB-DL",
},
want: Release{
TorrentName: "Servant.S01.2160p.ATVP.WEB-DL.DDP.5.1.Atmos.DV.HEVC-FLUX",
ReleaseTags: "MKV / 2160p / WEB-DL",
Title: "Servant",
Season: 1,
Episode: 0,
Resolution: "2160p",
Source: "WEB-DL",
Container: "mkv",
Codec: []string{"HEVC"},
Audio: []string{"DDP", "Atmos"},
AudioChannels: "5.1",
HDR: []string{"DV"},
Group: "FLUX",
Type: "series",
TorrentName: "Servant.S01.2160p.ATVP.WEB-DL.DDP.5.1.Atmos.DV.HEVC-FLUX",
NormalizedHash: "9b73e77d51fb0b69367ea96c761577b0",
ReleaseTags: "MKV / 2160p / WEB-DL",
Title: "Servant",
Season: 1,
Episode: 0,
Resolution: "2160p",
Source: "WEB-DL",
Container: "mkv",
Codec: []string{"HEVC"},
Audio: []string{"DDP", "Atmos"},
AudioChannels: "5.1",
HDR: []string{"DV"},
Group: "FLUX",
Website: "ATVP",
Type: rls.Series,
},
},
{
@ -86,20 +92,22 @@ func TestRelease_Parse(t *testing.T) {
ReleaseTags: "MKV | 2160p | WEB-DL",
},
want: Release{
TorrentName: "Servant.S01.2160p.ATVP.WEB-DL.DDP.5.1.Atmos.DV.HEVC-FLUX",
ReleaseTags: "MKV | 2160p | WEB-DL",
Title: "Servant",
Season: 1,
Episode: 0,
Resolution: "2160p",
Source: "WEB-DL",
Container: "mkv",
Codec: []string{"HEVC"},
Audio: []string{"DDP", "Atmos"},
AudioChannels: "5.1",
HDR: []string{"DV"},
Group: "FLUX",
Type: "series",
TorrentName: "Servant.S01.2160p.ATVP.WEB-DL.DDP.5.1.Atmos.DV.HEVC-FLUX",
NormalizedHash: "9b73e77d51fb0b69367ea96c761577b0",
ReleaseTags: "MKV | 2160p | WEB-DL",
Title: "Servant",
Season: 1,
Episode: 0,
Resolution: "2160p",
Source: "WEB-DL",
Container: "mkv",
Codec: []string{"HEVC"},
Audio: []string{"DDP", "Atmos"},
AudioChannels: "5.1",
HDR: []string{"DV"},
Group: "FLUX",
Website: "ATVP",
Type: rls.Series,
},
},
{
@ -109,20 +117,22 @@ func TestRelease_Parse(t *testing.T) {
ReleaseTags: "MP4 | 2160p | WEB-DL",
},
want: Release{
TorrentName: "Servant.S01.2160p.ATVP.WEB-DL.DDP.5.1.Atmos.DV.HEVC-FLUX",
ReleaseTags: "MP4 | 2160p | WEB-DL",
Title: "Servant",
Season: 1,
Episode: 0,
Resolution: "2160p",
Source: "WEB-DL",
Container: "mp4",
Codec: []string{"HEVC"},
Audio: []string{"DDP", "Atmos"},
AudioChannels: "5.1",
HDR: []string{"DV"},
Group: "FLUX",
Type: "series",
TorrentName: "Servant.S01.2160p.ATVP.WEB-DL.DDP.5.1.Atmos.DV.HEVC-FLUX",
NormalizedHash: "9b73e77d51fb0b69367ea96c761577b0",
ReleaseTags: "MP4 | 2160p | WEB-DL",
Title: "Servant",
Season: 1,
Episode: 0,
Resolution: "2160p",
Source: "WEB-DL",
Container: "mp4",
Codec: []string{"HEVC"},
Audio: []string{"DDP", "Atmos"},
AudioChannels: "5.1",
HDR: []string{"DV"},
Group: "FLUX",
Website: "ATVP",
Type: rls.Series,
},
},
{
@ -132,22 +142,24 @@ func TestRelease_Parse(t *testing.T) {
ReleaseTags: "MP4 | 2160p | WEB-DL | Freeleech!",
},
want: Release{
TorrentName: "Servant.S01.2160p.ATVP.WEB-DL.DDP.5.1.Atmos.DV.HEVC-FLUX",
ReleaseTags: "MP4 | 2160p | WEB-DL | Freeleech!",
Title: "Servant",
Season: 1,
Episode: 0,
Resolution: "2160p",
Source: "WEB-DL",
Container: "mp4",
Codec: []string{"HEVC"},
Audio: []string{"DDP", "Atmos"},
AudioChannels: "5.1",
HDR: []string{"DV"},
Group: "FLUX",
Freeleech: true,
Bonus: []string{"Freeleech"},
Type: "series",
TorrentName: "Servant.S01.2160p.ATVP.WEB-DL.DDP.5.1.Atmos.DV.HEVC-FLUX",
NormalizedHash: "9b73e77d51fb0b69367ea96c761577b0",
ReleaseTags: "MP4 | 2160p | WEB-DL | Freeleech!",
Title: "Servant",
Season: 1,
Episode: 0,
Resolution: "2160p",
Source: "WEB-DL",
Container: "mp4",
Codec: []string{"HEVC"},
Audio: []string{"DDP", "Atmos"},
AudioChannels: "5.1",
HDR: []string{"DV"},
Group: "FLUX",
Freeleech: true,
Bonus: []string{"Freeleech"},
Website: "ATVP",
Type: rls.Series,
},
},
{
@ -156,15 +168,16 @@ func TestRelease_Parse(t *testing.T) {
TorrentName: "Rippers.Revenge.2023.German.DL.1080p.BluRay.MPEG2-GROUP",
},
want: Release{
TorrentName: "Rippers.Revenge.2023.German.DL.1080p.BluRay.MPEG2-GROUP",
Title: "Rippers Revenge",
Year: 2023,
Language: []string{"GERMAN", "DL"},
Resolution: "1080p",
Source: "BluRay",
Codec: []string{"MPEG-2"},
Group: "GROUP",
Type: "movie",
TorrentName: "Rippers.Revenge.2023.German.DL.1080p.BluRay.MPEG2-GROUP",
NormalizedHash: "cb28c69ce117723d01ea6562ef3bae67",
Title: "Rippers Revenge",
Year: 2023,
Language: []string{"GERMAN", "DL"},
Resolution: "1080p",
Source: "BluRay",
Codec: []string{"MPEG-2"},
Group: "GROUP",
Type: rls.Movie,
},
},
{
@ -173,13 +186,14 @@ func TestRelease_Parse(t *testing.T) {
TorrentName: "Analogue.1080i.AHDTV.H264-ABCDEF",
},
want: Release{
TorrentName: "Analogue.1080i.AHDTV.H264-ABCDEF",
Title: "Analogue",
Resolution: "1080p", // rls does not differentiate between 1080i and 1080p which results in all 1080 releases being parsed as 1080p
Source: "AHDTV",
Codec: []string{"H.264"},
Group: "ABCDEF",
Type: "movie",
TorrentName: "Analogue.1080i.AHDTV.H264-ABCDEF",
NormalizedHash: "1a1eb25b15269d3189138326658aafe0",
Title: "Analogue",
Resolution: "1080p", // rls does not differentiate between 1080i and 1080p which results in all 1080 releases being parsed as 1080p
Source: "AHDTV",
Codec: []string{"H.264"},
Group: "ABCDEF",
Type: rls.Movie,
},
},
{
@ -189,17 +203,18 @@ func TestRelease_Parse(t *testing.T) {
ReleaseTags: "FLAC / Lossless / Log / 100% / Cue / CD",
},
want: Release{
TorrentName: "Artist - Albumname",
ReleaseTags: "FLAC / Lossless / Log / 100% / Cue / CD",
Title: "Artist",
Group: "Albumname",
Audio: []string{"Cue", "FLAC", "Lossless", "Log100", "Log"},
AudioFormat: "FLAC",
Source: "CD",
Bitrate: "Lossless",
HasLog: true,
LogScore: 100,
HasCue: true,
TorrentName: "Artist - Albumname",
NormalizedHash: "5de820bfae23f2bdc3a56c827a5230ac",
ReleaseTags: "FLAC / Lossless / Log / 100% / Cue / CD",
Title: "Artist",
Group: "Albumname",
Audio: []string{"Cue", "FLAC", "Lossless", "Log100", "Log"},
AudioFormat: "FLAC",
Source: "CD",
Bitrate: "Lossless",
HasLog: true,
LogScore: 100,
HasCue: true,
},
},
{
@ -210,14 +225,15 @@ func TestRelease_Parse(t *testing.T) {
ReleaseTags: "MP3 / 320 / Cassette",
},
want: Release{
TorrentName: "Various Artists - Music '21",
Tags: []string{"house, techno, tech.house, electro.house, future.house, bass.house, melodic.house"},
ReleaseTags: "MP3 / 320 / Cassette",
Title: "Various Artists - Music '21",
Source: "Cassette",
Audio: []string{"320", "MP3"},
AudioFormat: "MP3",
Bitrate: "320",
TorrentName: "Various Artists - Music '21",
NormalizedHash: "fbf7713532f4d2d03b62897fc990e37e",
Tags: []string{"house, techno, tech.house, electro.house, future.house, bass.house, melodic.house"},
ReleaseTags: "MP3 / 320 / Cassette",
Title: "Various Artists - Music '21",
Source: "Cassette",
Audio: []string{"320", "MP3"},
AudioFormat: "MP3",
Bitrate: "320",
},
},
{
@ -227,14 +243,15 @@ func TestRelease_Parse(t *testing.T) {
ReleaseTags: "MP3 / V0 (VBR) / CD",
},
want: Release{
TorrentName: "The artist (ザ・フリーダムユニティ) - Long album name",
ReleaseTags: "MP3 / V0 (VBR) / CD",
Title: "The artist",
Group: "name",
Source: "CD",
Audio: []string{"MP3", "VBR", "V0 (VBR)"},
AudioFormat: "MP3",
Bitrate: "V0 (VBR)",
TorrentName: "The artist (ザ・フリーダムユニティ) - Long album name",
NormalizedHash: "6b4051de7524f65d3c25be535d2c95ed",
ReleaseTags: "MP3 / V0 (VBR) / CD",
Title: "The artist",
Group: "name",
Source: "CD",
Audio: []string{"MP3", "VBR", "V0 (VBR)"},
AudioFormat: "MP3",
Bitrate: "V0 (VBR)",
},
},
{
@ -244,17 +261,18 @@ func TestRelease_Parse(t *testing.T) {
ReleaseTags: "FLAC / Lossless / Log / 100% / Cue / CD",
},
want: Release{
TorrentName: "Artist - Albumname",
ReleaseTags: "FLAC / Lossless / Log / 100% / Cue / CD",
Title: "Artist",
Group: "Albumname",
Audio: []string{"Cue", "FLAC", "Lossless", "Log100", "Log"},
AudioFormat: "FLAC",
Source: "CD",
Bitrate: "Lossless",
HasLog: true,
LogScore: 100,
HasCue: true,
TorrentName: "Artist - Albumname",
NormalizedHash: "5de820bfae23f2bdc3a56c827a5230ac",
ReleaseTags: "FLAC / Lossless / Log / 100% / Cue / CD",
Title: "Artist",
Group: "Albumname",
Audio: []string{"Cue", "FLAC", "Lossless", "Log100", "Log"},
AudioFormat: "FLAC",
Source: "CD",
Bitrate: "Lossless",
HasLog: true,
LogScore: 100,
HasCue: true,
},
},
{
@ -264,17 +282,18 @@ func TestRelease_Parse(t *testing.T) {
ReleaseTags: "FLAC / 24bit Lossless / Log / 100% / Cue / CD",
},
want: Release{
TorrentName: "Artist - Albumname",
ReleaseTags: "FLAC / 24bit Lossless / Log / 100% / Cue / CD",
Title: "Artist",
Group: "Albumname",
Audio: []string{"24BIT Lossless", "Cue", "FLAC", "Log100", "Log"},
AudioFormat: "FLAC",
Source: "CD",
Bitrate: "24BIT Lossless",
HasLog: true,
LogScore: 100,
HasCue: true,
TorrentName: "Artist - Albumname",
NormalizedHash: "5de820bfae23f2bdc3a56c827a5230ac",
ReleaseTags: "FLAC / 24bit Lossless / Log / 100% / Cue / CD",
Title: "Artist",
Group: "Albumname",
Audio: []string{"24BIT Lossless", "Cue", "FLAC", "Log100", "Log"},
AudioFormat: "FLAC",
Source: "CD",
Bitrate: "24BIT Lossless",
HasLog: true,
LogScore: 100,
HasCue: true,
},
},
{
@ -284,17 +303,18 @@ func TestRelease_Parse(t *testing.T) {
ReleaseTags: "FLAC / 24bit Lossless / Log / 78% / Cue / CD",
},
want: Release{
TorrentName: "Artist - Albumname",
ReleaseTags: "FLAC / 24bit Lossless / Log / 78% / Cue / CD",
Title: "Artist",
Group: "Albumname",
Audio: []string{"24BIT Lossless", "Cue", "FLAC", "Log78", "Log"},
AudioFormat: "FLAC",
Source: "CD",
Bitrate: "24BIT Lossless",
HasLog: true,
LogScore: 78,
HasCue: true,
TorrentName: "Artist - Albumname",
NormalizedHash: "5de820bfae23f2bdc3a56c827a5230ac",
ReleaseTags: "FLAC / 24bit Lossless / Log / 78% / Cue / CD",
Title: "Artist",
Group: "Albumname",
Audio: []string{"24BIT Lossless", "Cue", "FLAC", "Log78", "Log"},
AudioFormat: "FLAC",
Source: "CD",
Bitrate: "24BIT Lossless",
HasLog: true,
LogScore: 78,
HasCue: true,
},
},
{
@ -303,18 +323,22 @@ func TestRelease_Parse(t *testing.T) {
TorrentName: "I Am Movie 2007 Theatrical UHD BluRay 2160p DTS-HD MA 5.1 DV HEVC HYBRID REMUX-GROUP1",
},
want: Release{
TorrentName: "I Am Movie 2007 Theatrical UHD BluRay 2160p DTS-HD MA 5.1 DV HEVC HYBRID REMUX-GROUP1",
Title: "I Am Movie",
Resolution: "2160p",
Source: "UHD.BluRay",
Codec: []string{"HEVC"},
HDR: []string{"DV"},
Audio: []string{"DTS-HD.MA"},
AudioChannels: "5.1",
Year: 2007,
Group: "GROUP1",
Other: []string{"HYBRiD", "REMUX"},
Type: "movie",
TorrentName: "I Am Movie 2007 Theatrical UHD BluRay 2160p DTS-HD MA 5.1 DV HEVC HYBRID REMUX-GROUP1",
NormalizedHash: "f70840f42149d7d2b539c75f5e853493",
Title: "I Am Movie",
Resolution: "2160p",
Source: "UHD.BluRay",
Codec: []string{"HEVC"},
HDR: []string{"DV"},
Audio: []string{"DTS-HD.MA"},
AudioChannels: "5.1",
Year: 2007,
Group: "GROUP1",
Hybrid: true,
Cut: []string{"Theatrical.Cut"},
MediaProcessing: "REMUX",
Other: []string{"HYBRiD", "REMUX"},
Type: rls.Movie,
},
},
{
@ -323,19 +347,21 @@ func TestRelease_Parse(t *testing.T) {
TorrentName: "The Peripheral (2022) S01 (2160p AMZN WEB-DL H265 HDR10+ DDP 5.1 English - GROUP1)",
},
want: Release{
TorrentName: "The Peripheral (2022) S01 (2160p AMZN WEB-DL H265 HDR10+ DDP 5.1 English - GROUP1)",
Title: "The Peripheral",
Resolution: "2160p",
Source: "WEB-DL",
Codec: []string{"H.265"},
HDR: []string{"HDR10+"},
Audio: []string{"DDP"},
AudioChannels: "5.1",
Year: 2022,
Group: "GROUP1",
Season: 1,
Language: []string{"ENGLiSH"},
Type: "series",
TorrentName: "The Peripheral (2022) S01 (2160p AMZN WEB-DL H265 HDR10+ DDP 5.1 English - GROUP1)",
NormalizedHash: "33fe914c00ea5a2945883cffb4183923",
Title: "The Peripheral",
Resolution: "2160p",
Source: "WEB-DL",
Codec: []string{"H.265"},
HDR: []string{"HDR10+"},
Audio: []string{"DDP"},
AudioChannels: "5.1",
Year: 2022,
Group: "GROUP1",
Season: 1,
Language: []string{"ENGLiSH"},
Website: "AMZN",
Type: rls.Series,
},
},
{
@ -344,18 +370,20 @@ func TestRelease_Parse(t *testing.T) {
TorrentName: "Old Movie 1954 2160p Remux DoVi HDR10 HEVC DTS-HD MA 5.1-CiNEPHiLES",
},
want: Release{
TorrentName: "Old Movie 1954 2160p Remux DoVi HDR10 HEVC DTS-HD MA 5.1-CiNEPHiLES",
Title: "Old Movie",
Year: 1954,
Source: "UHD.BluRay",
Resolution: "2160p",
Other: []string{"REMUX"},
HDR: []string{"DV", "HDR10"},
Codec: []string{"HEVC"},
Audio: []string{"DTS-HD.MA"},
AudioChannels: "5.1",
Group: "CiNEPHiLES",
Type: "movie",
TorrentName: "Old Movie 1954 2160p Remux DoVi HDR10 HEVC DTS-HD MA 5.1-CiNEPHiLES",
Title: "Old Movie",
NormalizedHash: "9990678babd4f24f268dfe7856b29f0e",
Year: 1954,
Source: "UHD.BluRay",
Resolution: "2160p",
Other: []string{"REMUX"},
HDR: []string{"DV", "HDR10"},
Codec: []string{"HEVC"},
Audio: []string{"DTS-HD.MA"},
AudioChannels: "5.1",
Group: "CiNEPHiLES",
Type: rls.Movie,
MediaProcessing: "REMUX",
},
},
{
@ -364,17 +392,19 @@ func TestRelease_Parse(t *testing.T) {
TorrentName: "Death Hunt 1981 1080p Remux AVC DTS-HD MA 2.0-playBD",
},
want: Release{
TorrentName: "Death Hunt 1981 1080p Remux AVC DTS-HD MA 2.0-playBD",
Title: "Death Hunt",
Year: 1981,
Source: "BluRay",
Resolution: "1080p",
Other: []string{"REMUX"},
Codec: []string{"AVC"},
Audio: []string{"DTS-HD.MA"},
AudioChannels: "2.0",
Group: "playBD",
Type: "movie",
TorrentName: "Death Hunt 1981 1080p Remux AVC DTS-HD MA 2.0-playBD",
Title: "Death Hunt",
NormalizedHash: "e6b1b708b8ea39470de6d327da2920d6",
Year: 1981,
Source: "BluRay",
Resolution: "1080p",
Other: []string{"REMUX"},
Codec: []string{"AVC"},
Audio: []string{"DTS-HD.MA"},
AudioChannels: "2.0",
Group: "playBD",
Type: rls.Movie,
MediaProcessing: "REMUX",
},
},
{
@ -383,18 +413,20 @@ func TestRelease_Parse(t *testing.T) {
TorrentName: "Old Movie 1954 2160p Remux DoVi HDR10 HEVC DTS-HD MA 5.1-VHS",
},
want: Release{
TorrentName: "Old Movie 1954 2160p Remux DoVi HDR10 HEVC DTS-HD MA 5.1-VHS",
Title: "Old Movie",
Year: 1954,
Source: "UHD.BluRay",
Resolution: "2160p",
Other: []string{"REMUX"},
HDR: []string{"DV", "HDR10"},
Codec: []string{"HEVC"},
Audio: []string{"DTS-HD.MA"},
AudioChannels: "5.1",
Group: "VHS",
Type: "movie",
TorrentName: "Old Movie 1954 2160p Remux DoVi HDR10 HEVC DTS-HD MA 5.1-VHS",
Title: "Old Movie",
NormalizedHash: "1ea814c8c21979e7692efb1b24298e95",
Year: 1954,
Source: "UHD.BluRay",
Resolution: "2160p",
Other: []string{"REMUX"},
HDR: []string{"DV", "HDR10"},
Codec: []string{"HEVC"},
Audio: []string{"DTS-HD.MA"},
AudioChannels: "5.1",
Group: "VHS",
Type: rls.Movie,
MediaProcessing: "REMUX",
},
},
{
@ -403,18 +435,20 @@ func TestRelease_Parse(t *testing.T) {
TorrentName: "Old Movie 1954 2160p Remux DoVi HDR10 HEVC DTS-HD MA 5.1 VHS",
},
want: Release{
TorrentName: "Old Movie 1954 2160p Remux DoVi HDR10 HEVC DTS-HD MA 5.1 VHS",
Title: "Old Movie",
Year: 1954,
Source: "UHD.BluRay",
Resolution: "2160p",
Other: []string{"REMUX"},
HDR: []string{"DV", "HDR10"},
Codec: []string{"HEVC"},
Audio: []string{"DTS-HD.MA"},
AudioChannels: "5.1",
Group: "VHS",
Type: "movie",
TorrentName: "Old Movie 1954 2160p Remux DoVi HDR10 HEVC DTS-HD MA 5.1 VHS",
Title: "Old Movie",
NormalizedHash: "1ea814c8c21979e7692efb1b24298e95",
Year: 1954,
Source: "UHD.BluRay",
Resolution: "2160p",
Other: []string{"REMUX"},
HDR: []string{"DV", "HDR10"},
Codec: []string{"HEVC"},
Audio: []string{"DTS-HD.MA"},
AudioChannels: "5.1",
Group: "VHS",
Type: rls.Movie,
MediaProcessing: "REMUX",
},
},
}
@ -863,3 +897,220 @@ func Test_getUniqueTags(t *testing.T) {
})
}
}
func TestRelease_Hash(t *testing.T) {
type fields struct {
TorrentName string
}
tests := []struct {
name string
fields fields
want string
}{
{
name: "1",
fields: fields{
TorrentName: "That.Movie.2023.2160p.BluRay.DTS-HD.5.1.x265-GROUP",
},
want: "833e6c4c78e960ccbce4ef68f6564082",
},
{
name: "2",
fields: fields{
TorrentName: "That Movie 2023 2160p BluRay DTS-HD 5.1 x265-GROUP",
},
want: "833e6c4c78e960ccbce4ef68f6564082",
},
{
name: "3",
fields: fields{
TorrentName: "That Movie 2023 2160p BluRay DTS-HD 5 1 x265-GROUP",
},
want: "833e6c4c78e960ccbce4ef68f6564082",
},
{
name: "4",
fields: fields{
TorrentName: "That Movie 2023 1080p Remux DTS-HD 5 1 AVC-GROUP",
},
want: "ee15fe3c926a5b40370aec970f6b4c43",
},
{
name: "5",
fields: fields{
TorrentName: "That Movie 2023 Bluray 1080p Remux DTS-HD 5.1 AVC-GROUP",
},
want: "ee15fe3c926a5b40370aec970f6b4c43",
},
{
name: "6",
fields: fields{
TorrentName: "That Movie 2023 Theatrical Cut Bluray 1080p Remux DTS-HD 5.1 AVC-GROUP",
},
want: "db7e55fb3471e76ba27bf8a2e1e59236",
},
{
name: "7",
fields: fields{
TorrentName: "That Movie 2023 Directors Cut Bluray 1080p Remux DTS-HD 5.1 AVC-GROUP",
},
want: "bccfe6762b34cbc3a54784351b5424fd",
},
{
name: "8",
fields: fields{
TorrentName: "That Show 2023 S01 Bluray 1080p Remux DTS-HD 5.1 AVC-GROUP",
},
want: "d1c74c33e1c36f3545d06aa8857610b1",
},
{
name: "9",
fields: fields{
TorrentName: "That Show 2023 S01 Bluray 1080p Remux DTS-HD 5.1 AVC-GROUP2",
},
want: "29e228f705f070041e72ebaa7b29239e",
},
{
name: "10",
fields: fields{
TorrentName: "That Show 2023 S1 Bluray 1080p Remux DTS-HD 5.1 AVC-GROUP2",
},
want: "29e228f705f070041e72ebaa7b29239e",
},
{
name: "11",
fields: fields{
TorrentName: "That Show 2023 S2 Bluray 1080p Remux DTS-HD 5.1 AVC-GROUP2",
},
want: "cbc67f4eec93428361eddf040654e077",
},
{
name: "12",
fields: fields{
TorrentName: "That Show 2023 S01E01 2160p WEB-DL DTS-HD 5.1 HEVC-GROUP",
},
want: "806240c76cc32413efd5bcf033abf512",
},
{
name: "13",
fields: fields{
TorrentName: "That Show 2023 S01E01 This is the start 2160p WEB-DL DTS-HD 5.1 HEVC-GROUP",
},
want: "33c866b898a59d255a09c10bd23a5e3f",
},
{
name: "14",
fields: fields{
TorrentName: "That Show 2023 REPACK S01E01 This is the start 2160p WEB-DL DTS-HD 5.1 HEVC-GROUP",
},
want: "7be5ebeee948820f53f24091054fc7cd",
},
{
name: "15",
fields: fields{
TorrentName: "That Show 2023 REPACK2 S01E01 This is the start 2160p WEB-DL DTS-HD 5.1 HEVC-GROUP",
},
want: "84e3e70f6c983f96f10cba50213a338d",
},
{
name: "16",
fields: fields{
TorrentName: "That Show 2023 PROPER S01E01 This is the start 2160p WEB-DL DTS-HD 5.1 HEVC-GROUP",
},
want: "52d5d1a28a340bc5a836c0244c206c3b",
},
{
name: "17",
fields: fields{
TorrentName: "That Show 2023 S01E01 This is the start 2160p WEB-DL DV DTS-HD 5.1 HEVC-GROUP",
},
want: "5e196039d426649ab5221c3e87ed108d",
},
{
name: "18",
fields: fields{
TorrentName: "That Show 2023 S01E01 This is the start 2160p WEB-DL DoVi DTS-HD 5.1 HEVC-GROUP",
},
want: "5e196039d426649ab5221c3e87ed108d",
},
{
name: "19",
fields: fields{
TorrentName: "That Show 2023 S01E01 This is the start 2160p WEB-DL DoVi HDR10 DTS-HD 5.1 HEVC-GROUP",
},
want: "f52a9a2f26e94c90bdaa8eaca2136cc3",
},
{
name: "20",
fields: fields{
TorrentName: "That Show 2023 S01E01 This is the start 2160p WEB-DL DV HDR10 DTS-HD 5.1 HEVC-GROUP",
},
want: "f52a9a2f26e94c90bdaa8eaca2136cc3",
},
{
name: "21",
fields: fields{
TorrentName: "that show 2023 S01E01 this is the start 2160p WEB-DL DV HDR10 DTS-HD 5.1 HEVC-GROUP",
},
want: "f52a9a2f26e94c90bdaa8eaca2136cc3",
},
{
name: "22",
fields: fields{
TorrentName: "That Show 2023 S01E01 This is the start 2160p HYBRID WEB-DL DV HDR10 DTS-HD 5.1 HEVC-GROUP",
},
want: "414f792a345148ed4c8e070ef77340ff",
},
{
name: "23",
fields: fields{
TorrentName: "That Movie 2008 1080p GBR Blu-ray AVC TrueHD 5.1",
},
want: "70bb23ff5da4209419b6d83628310a43",
},
{
name: "24",
fields: fields{
TorrentName: "That Movie 2008 1080p GER Blu-ray AVC TrueHD 5.1",
},
want: "369ba9f92fb7819e0d07791fddf3c85f",
},
{
name: "25",
fields: fields{
TorrentName: "That Show 2023 PROPER S01E01 This is the start 2160p AMZN WEB-DL DTS-HD 5.1 HEVC-GROUP",
},
want: "28e215fee790fbad3957383162f7c5a6",
},
{
name: "26",
fields: fields{
TorrentName: "That Show 2023 PROPER S01E01 This is the start 2160p DSNP WEB-DL DTS-HD 5.1 HEVC-GROUP",
},
want: "6a26cd652170d0efc4b0e0320fbd38c1",
},
{
name: "27",
fields: fields{
TorrentName: "That Other Show 2023 S01E01 2160p WEB-DL DV HDR10 DTS-HD 5.1 HEVC-GROUP",
},
want: "544c2fc4024ece5664c0f623a5feaeab",
},
{
name: "28",
fields: fields{
TorrentName: "That Other Show 2023 S01E01 2160p WEB-DL DV HDR10+ DTS-HD 5.1 HEVC-GROUP",
},
want: "63b5d87abe5fb49131785de426708d31",
},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
r := &Release{
TorrentName: tt.fields.TorrentName,
}
r.ParseString(tt.fields.TorrentName)
assert.Equalf(t, tt.want, r.NormalizedHash, "Hash()")
})
}
}

5
internal/domain/releasetags.go
View file

@ -425,9 +425,14 @@ func ParseReleaseTags(tags []string) ReleaseTags {
return releaseTags
}
func ParseReleaseTagString(tags string) ReleaseTags {
releaseTags := ReleaseTags{}
if tags == "" {
return releaseTags
}
for tagType, tagInfos := range types {
for _, info := range tagInfos {