Update ImagePipe (#705)

* Update ImagePipe, fixes music finder

* Change rodio library, enable by default checking for broken audio files

* Don't crash when checking for broken files

* Fix stopping of scan in broken files

* No more

* PDF Support

* Infer and crash handler for music

.github/workflows/linux.yml

@@ -130,7 +130,7 @@ jobs:
   linux-gui:
     strategy:
       matrix:
-        toolchain: [ stable, 1.57.0 ]
+        toolchain: [ stable, 1.60.0 ]
         type: [ release ]
     runs-on: ubuntu-20.04
     steps:

.github/workflows/quality.yml

@@ -10,7 +10,7 @@ env:
 
 jobs:
   quality:
-    runs-on: ubuntu-latest
+    runs-on: ubuntu-22.04
     steps:
       - uses: actions/checkout@v2
 

README.md

@@ -138,7 +138,7 @@ You can help by creating:
 - Bug reports - memory leaks, unexpected behavior, crashes
 - Feature proposals - proposal to change/add/delete some features
 - Pull Requests - implementing a new feature yourself or fixing bugs.
-  If the change is bigger, then it's a good idea to open a new issue to discuss changes.
+  If the change is bigger, then it's a good idea to open a new issue to discuss changes, but issues with label `PR welcome` are already checked and accepted.
 - Documentation - There is an [instruction](instructions/Instruction.md) which you can improve.
 - Translations - Instruction how to translate files is available [here](instructions/Translations.md)
 
@@ -178,5 +178,7 @@ Big thanks to Pádraig Brady, creator of fantastic FSlint, because without his w
 
 Thanks also to all the people who create patches for this program, make it available on other systems, create videos, articles about it etc.
 
+Also I really appreciate work of people that create crates on which Czkawka is based and for that I try to report bugs to make it even better.
+
 ## Donations
 If you are using the app, I would appreciate a donation for its further development, which can be done [here](https://github.com/sponsors/qarmin).

czkawka_core/Cargo.toml

@@ -20,24 +20,25 @@ directories-next = "2.0.0"
 # Needed by similar images
 image_hasher = "1.0.0"
 bk-tree = "0.4.0"
-image = "0.24.1"
+image = "0.24.2"
 hamming = "0.1.3"
 
 # Needed by same music
 bitflags = "1.3.2"
-lofty="0.6.0"
+lofty="0.6.2"
 
 # Futures - needed by async progress sender
 futures = "0.3.21"
 
 # Needed by broken files
 zip = { version = "0.6.2", features=["aes-crypto", "bzip2", "deflate", "time"], default-features = false}
-rodio = { version = "0.15.0", optional = true }
+audio_checker = "0.1.0"
+pdf = "0.7.2"
 
 # Hashes for duplicate files
 blake3 = "1.3.1"
 crc32fast = "1.3.2"
-xxhash-rust = { version = "0.8.4", features = ["xxh3"] }
+xxhash-rust = { version = "0.8.5", features = ["xxh3"] }
 
 tempfile = "3.3.0"
 
@@ -46,25 +47,20 @@ vid_dup_finder_lib = "0.1.0"
 ffmpeg_cmdline_utils = "0.1.1"
 
 # Saving/Loading Cache
-serde = "1.0.136"
+serde = "1.0.137"
 bincode = "1.3.3"
-serde_json = "1.0.79"
+serde_json = "1.0.81"
 
 # Language
 i18n-embed = { version = "0.13.4", features = ["fluent-system", "desktop-requester"] }
 i18n-embed-fl = "0.6.4"
-rust-embed = "6.3.0"
+rust-embed = "6.4.0"
 once_cell = "1.10.0"
 
 # Raw image files
-rawloader = "0.37.0"
-imagepipe = "0.4.0"
+rawloader = "0.37.1"
+imagepipe = "0.5.0"
 
 # Checking for invalid extensions
 mime_guess = "2.0.4"
-infer = "0.7.0"
-
-[features]
-default = []
-
-broken_audio = ["rodio"]
+infer = "0.8.0"

czkawka_core/src/bad_extensions.rs

@@ -21,6 +21,8 @@ use crate::common_items::ExcludedItems;
 use crate::common_messages::Messages;
 use crate::common_traits::*;
 
+static DISABLED_EXTENSIONS: &[&str] = &["file", "cache", "bak"]; // Such files can have any type inside
+
 // This adds several workarounds for bugs/invalid recognizing types by external libraries
 // ("real_content_extension", "current_file_extension")
 static WORKAROUNDS: &[(&str, &str)] = &[
@@ -43,9 +45,14 @@ static WORKAROUNDS: &[(&str, &str)] = &[
     ("exe", "sys"),
     ("exe", "tlb"),
     ("exe", "vxd"),
-    // Other
-    ("exe", "doc"), // Not sure whe docx is not recognized
     ("exe", "sys"),
+    ("exe", "mod16"),
+    // Other
+    ("zip", "odg"), // Libreoffice
+    ("ods", "ots"), // Libreoffice
+    ("exe", "efi"),
+    ("sh", "sample"), // Git
+    ("exe", "signed"),
     ("gz", "blend"),
     ("gz", "crate"),
     ("gz", "svgz"),
@@ -53,16 +60,28 @@ static WORKAROUNDS: &[(&str, &str)] = &[
     ("html", "md"),
     ("html", "svg"), // Quite strange, but yes it works
     ("jpg", "jfif"),
+    ("mobi", "azw3"),
     ("obj", "o"),
+    ("obj", "bin"),
     ("odp", "otp"),
     ("odt", "ott"),
     ("ogg", "ogv"),
     ("pptx", "ppsx"),
     ("sh", "bash"),
     ("sh", "py"),
+    ("sh", "pyx"),
+    ("xml", "sopcinfo"), // Quartus
+    ("xml", "bsp"),      // Quartus
+    ("xml", "fb2"),
+    ("xml", "user"), // Qtcreator
     ("sh", "rs"),
+    ("sh", "pl"),   // Gnome/Linux
+    ("sh", "pm"),   // Gnome/Linux
+    ("xml", "cbp"), // CodeBlocks config
     ("xml", "cmb"),
+    ("xml", "cfg"),
     ("xml", "conf"),
+    ("xml", "config"),
     ("xml", "dae"),
     ("xml", "docbook"),
     ("xml", "gir"),
@@ -71,16 +90,29 @@ static WORKAROUNDS: &[(&str, &str)] = &[
     ("xml", "kdenlive"),
     ("xml", "lang"),
     ("xml", "svg"),
-    ("xml", "ui"),
-    ("xml", "vcproj"),
+    ("xml", "ui"),     // Cambalache, Glade
+    ("xml", "vcproj"), // VisualStudio
+    ("xml", "iml"),    // Intelij Idea
+    ("xml", "manifest"),
+    ("xml", "xcd"), // Libreoffice files
+    ("xml", "policy"),
+    ("xml", "qsys"), // Quartus
+    ("xml", "xba"),  // Libreoffice
     ("zip", "apk"),
+    ("zip", "doc"),
+    ("zip", "docx"),
     ("zip", "dat"),
-    ("zip", "jar"),
-    ("zip", "kra"),
+    ("zip", "jar"), // Java
+    ("zip", "kra"), // Krita
     ("zip", "nupkg"),
     ("zip", "pptx"),
     ("zip", "whl"),
     ("zip", "xpi"),
+    ("zip", "zcos"),
+    ("zip", "cbr"), // Komiksy
+    // Probably bug in external library
+    ("exe", "doc"), // Not sure whe doc is not recognized
+    ("exe", "xls"), // Not sure whe xls is not recognized
 ];
 
 #[derive(Clone)]
@@ -285,8 +317,6 @@ impl BadExtensions {
                     return None;
                 }
 
-                let current_extension;
-
                 // Check what exactly content file contains
                 let kind = match infer::get_from_path(&file_entry.path) {
                     Ok(k) => match k {
@@ -298,8 +328,12 @@ impl BadExtensions {
                 let proper_extension = kind.extension();
 
                 // Extract current extension from file
+                let current_extension;
                 if let Some(extension) = file_entry.path.extension() {
                     let extension = extension.to_string_lossy().to_lowercase();
+                    if DISABLED_EXTENSIONS.contains(&extension.as_str()) {
+                        return Some(None);
+                    }
                     // Text longer than 10 characters is not considered as extension
                     if extension.len() > 10 {
                         current_extension = "".to_string();
@@ -335,7 +369,7 @@ impl BadExtensions {
                             }
                         }
 
-                        let mut guessed_multiple_extensions = "".to_string();
+                        let mut guessed_multiple_extensions = format!("({}) - ", proper_extension);
                         for ext in &all_available_extensions {
                             guessed_multiple_extensions.push_str(ext);
                             guessed_multiple_extensions.push(',');

czkawka_core/src/broken_files.rs

@@ -10,10 +10,13 @@ use std::time::{Duration, SystemTime, UNIX_EPOCH};
 use std::{fs, mem, panic, thread};
 
 use crossbeam_channel::Receiver;
+use pdf::PdfError;
+use pdf::PdfError::Try;
 use rayon::prelude::*;
 use serde::{Deserialize, Serialize};
 
-use crate::common::{open_cache_folder, Common, LOOP_DURATION};
+use crate::common::{open_cache_folder, Common, LOOP_DURATION, PDF_FILES_EXTENSIONS};
+use crate::common::{AUDIO_FILES_EXTENSIONS, IMAGE_RS_BROKEN_FILES_EXTENSIONS, ZIP_FILES_EXTENSIONS};
 use crate::common_directory::Directories;
 use crate::common_extensions::Extensions;
 use crate::common_items::ExcludedItems;
@@ -21,7 +24,6 @@ use crate::common_messages::Messages;
 use crate::common_traits::*;
 use crate::flc;
 use crate::localizer_core::generate_translation_hashmap;
-use crate::similar_images::{AUDIO_FILES_EXTENSIONS, IMAGE_RS_BROKEN_FILES_EXTENSIONS, ZIP_FILES_EXTENSIONS};
 
 #[derive(Debug)]
 pub struct ProgressData {
@@ -51,8 +53,8 @@ pub enum TypeOfFile {
     Unknown = -1,
     Image = 0,
     ArchiveZip,
-    #[cfg(feature = "broken_audio")]
     Audio,
+    PDF,
 }
 
 /// Info struck with helpful information's about results
@@ -296,7 +298,7 @@ impl BrokenFiles {
                                 continue 'dir;
                             }
 
-                            let type_of_file = check_extension_avaibility(&file_name_lowercase);
+                            let type_of_file = check_extension_availability(&file_name_lowercase);
                             if type_of_file == TypeOfFile::Unknown {
                                 continue 'dir;
                             }
@@ -392,8 +394,6 @@ impl BrokenFiles {
         }
 
         //// PROGRESS THREAD START
-        let check_was_stopped = AtomicBool::new(false); // Used for breaking from GUI and ending check thread
-
         let progress_thread_run = Arc::new(AtomicBool::new(true));
         let atomic_file_counter = Arc::new(AtomicUsize::new(0));
 
@@ -425,61 +425,101 @@ impl BrokenFiles {
             .map(|(_, mut file_entry)| {
                 atomic_file_counter.fetch_add(1, Ordering::Relaxed);
                 if stop_receiver.is_some() && stop_receiver.unwrap().try_recv().is_ok() {
-                    check_was_stopped.store(true, Ordering::Relaxed);
                     return None;
                 }
 
                 match file_entry.type_of_file {
                     TypeOfFile::Image => {
-                        let file_entry_clone = file_entry.clone();
+                        let mut file_entry_clone = file_entry.clone();
 
                         let result = panic::catch_unwind(|| {
-                            match image::open(&file_entry.path) {
-                                Ok(_) => Some(None),
-                                Err(t) => {
-                                    let error_string = t.to_string();
-                                    // This error is a problem with image library, remove check when https://github.com/image-rs/jpeg-decoder/issues/130 will be fixed
-                                    if !error_string.contains("spectral selection is not allowed in non-progressive scan") {
-                                        file_entry.error_string = error_string;
-                                        Some(Some(file_entry))
-                                    } else {
-                                        Some(None)
-                                    }
+                            if let Err(e) = image::open(&file_entry.path) {
+                                let error_string = e.to_string();
+                                // This error is a problem with image library, remove check when https://github.com/image-rs/jpeg-decoder/issues/130 will be fixed
+                                if error_string.contains("spectral selection is not allowed in non-progressive scan") {
+                                    return Some(None);
                                 }
+                                file_entry.error_string = error_string;
                             }
+                            Some(Some(file_entry))
                         });
 
-                        // If image crashed during opening, we just skip checking its hash and go on
+                        // If image crashed during opening, needs to be printed info about crashes thing
                         if let Ok(image_result) = result {
-                             image_result
+                            image_result
                         } else {
                             println!("Image-rs library crashed when opening \"{:?}\" image, please check if problem happens with latest image-rs version(this can be checked via https://github.com/qarmin/ImageOpening tool) and if it is not reported, please report bug here - https://github.com/image-rs/image/issues", file_entry_clone.path);
-                             Some(Some(file_entry_clone))
+                            file_entry_clone.error_string = "Image crashes due parsing, please check if problem happens with updated https://github.com/qarmin/ImageOpening and later report here https://github.com/image-rs/image/issues".to_string();
+                            Some(Some(file_entry_clone))
                         }
                     }
                     TypeOfFile::ArchiveZip => match fs::File::open(&file_entry.path) {
-                        Ok(file) => match zip::ZipArchive::new(file) {
-                            Ok(_) => Some(None),
-                            Err(e) => {
-                                // TODO Maybe filter out unnecessary types of errors
+                        Ok(file) => {
+                            if let Err(e) = zip::ZipArchive::new(file) {
                                 file_entry.error_string = e.to_string();
-                                Some(Some(file_entry))
                             }
+                            Some(Some(file_entry))
                         },
-                        Err(_inspected) => Some(None), // TODO maybe throw error or something
+                        Err(_inspected) => Some(None)
                     },
-                    #[cfg(feature = "broken_audio")]
                     TypeOfFile::Audio => match fs::File::open(&file_entry.path) {
-                        Ok(file) => match rodio::Decoder::new(BufReader::new(file)) {
-                            Ok(_) => Some(None),
-                            Err(e) => {
-                                file_entry.error_string = e.to_string();
-                                Some(Some(file_entry))
-                            }
-                        },
+                        Ok(file) =>
+                            {
+                                let mut file_entry_clone = file_entry.clone();
+
+                                let result = panic::catch_unwind(|| {
+                                    if let Err(e) = audio_checker::parse_audio_file(file) {
+                                        file_entry.error_string = e.to_string();
+                                    }
+                                    Some(Some(file_entry))
+                                });
+
+                                if let Ok(audio_result) = result {
+                                    audio_result
+                                } else {
+                                    println!("External parsing audio library crashed when opening \"{:?}\" audio file, please report bug here - https://github.com/qarmin/audio_checker/issues", file_entry_clone.path);
+                                    file_entry_clone.error_string = "Audio crashes due parsing, please report bug here - https://github.com/qarmin/audio_checker/issues".to_string();
+                                    Some(Some(file_entry_clone))
+                                }
+                            },
                         Err(_inspected) => Some(None), // TODO maybe throw error or something
                     },
 
+                    TypeOfFile::PDF => {
+                        match fs::read(&file_entry.path) {
+                            Ok(content) => {
+                                // Will be available in pdf > 0.7.2
+                                // let parser_options = ParseOptions {
+                                //     allow_error_in_option: true,
+                                //     allow_xref_error: true,
+                                //     allow_invalid_ops: true,
+                                //     allow_missing_endobj: true,
+                                // };
+                                // if let Err(e) = pdf::file::File::from_data_with_options(content, parser_options) {
+
+                                let mut file_entry_clone = file_entry.clone();
+                                let result = panic::catch_unwind(|| {
+                                    if let Err(e) = pdf::file::File::from_data(content) {
+                                        file_entry.error_string = e.to_string();
+                                        let error = unpack_pdf_error(e);
+                                        if let pdf::PdfError::InvalidPassword = error {
+                                            return Some(None);
+                                        }
+                                    }
+                                    Some(Some(file_entry))
+                                });
+                                if let Ok(pdf_result) = result {
+                                    pdf_result
+                                } else {
+                                    println!("PDF-rs library crashed when opening \"{:?}\" pdf, and if it is not reported, please report bug here - https://github.com/pdf-rs/pdf", file_entry_clone.path);
+                                    file_entry_clone.error_string = "PDF-rs library crashed when opening pdf, and if it is not reported, please report bug here - https://github.com/pdf-rs/pdf".to_string();
+                                    Some(Some(file_entry_clone))
+                                }
+                            },
+                            Err(_inspected) => Some(None)
+                        }
+                    }
+
                     // This means that cache read invalid value because maybe cache comes from different czkawka version
                     TypeOfFile::Unknown => Some(None),
                 }
@@ -500,7 +540,7 @@ impl BrokenFiles {
 
         if self.use_cache {
             // Must save all results to file, old loaded from file with all currently counted results
-            let mut all_results: BTreeMap<String, FileEntry> = self.files_to_check.clone();
+            let mut all_results: BTreeMap<String, FileEntry> = Default::default();
 
             for file_entry in vec_file_entry.clone() {
                 all_results.insert(file_entry.path.to_string_lossy().to_string(), file_entry);
@@ -511,11 +551,6 @@ impl BrokenFiles {
             save_cache_to_file(&all_results, &mut self.text_messages, self.save_also_as_json);
         }
 
-        // Break if stop was clicked after saving to cache
-        if check_was_stopped.load(Ordering::Relaxed) {
-            return false;
-        }
-
         self.broken_files = vec_file_entry
             .into_par_iter()
             .filter_map(|f| if f.error_string.is_empty() { None } else { Some(f) })
@@ -717,22 +752,31 @@ fn get_cache_file() -> String {
     "cache_broken_files.bin".to_string()
 }
 
-fn check_extension_avaibility(file_name_lowercase: &str) -> TypeOfFile {
+fn check_extension_availability(file_name_lowercase: &str) -> TypeOfFile {
     if IMAGE_RS_BROKEN_FILES_EXTENSIONS.iter().any(|e| file_name_lowercase.ends_with(e)) {
         TypeOfFile::Image
     } else if ZIP_FILES_EXTENSIONS.iter().any(|e| file_name_lowercase.ends_with(e)) {
         TypeOfFile::ArchiveZip
     } else if AUDIO_FILES_EXTENSIONS.iter().any(|e| file_name_lowercase.ends_with(e)) {
-        #[cfg(feature = "broken_audio")]
-        {
-            TypeOfFile::Audio
-        }
-
-        #[cfg(not(feature = "broken_audio"))]
-        {
-            TypeOfFile::Unknown
-        }
+        TypeOfFile::Audio
+    } else if PDF_FILES_EXTENSIONS.iter().any(|e| file_name_lowercase.ends_with(e)) {
+        TypeOfFile::PDF
     } else {
         TypeOfFile::Unknown
     }
 }
+
+fn unpack_pdf_error(e: PdfError) -> PdfError {
+    if let Try {
+        file: _,
+        line: _,
+        column: _,
+        // context: _,
+        source,
+    } = e
+    {
+        unpack_pdf_error(*source)
+    } else {
+        e
+    }
+}

czkawka_core/src/common.rs

@@ -9,6 +9,36 @@ use std::path::{Path, PathBuf};
 use std::time::SystemTime;
 
 /// Class for common functions used across other class/functions
+pub const RAW_IMAGE_EXTENSIONS: &[&str] = &[
+    ".mrw", ".arw", ".srf", ".sr2", ".mef", ".orf", ".srw", ".erf", ".kdc", ".kdc", ".dcs", ".rw2", ".raf", ".dcr", ".dng", ".pef", ".crw", ".iiq", ".3fr", ".nrw", ".nef", ".mos",
+    ".cr2", ".ari",
+];
+pub const IMAGE_RS_EXTENSIONS: &[&str] = &[
+    ".jpg", ".jpeg", ".png", ".bmp", ".tiff", ".tif", ".tga", ".ff", ".jif", ".jfi", ".webp", ".gif", ".ico", ".exr", ".hdr", "dds",
+];
+
+pub const IMAGE_RS_SIMILAR_IMAGES_EXTENSIONS: &[&str] = &[
+    ".jpg", ".jpeg", ".png", ".tiff", ".tif", ".tga", ".ff", ".jif", ".jfi", ".bmp", ".webp", ".exr", ".hdr", "dds",
+];
+
+pub const IMAGE_RS_BROKEN_FILES_EXTENSIONS: &[&str] = &[
+    ".jpg", ".jpeg", ".png", ".tiff", ".tif", ".tga", ".ff", ".jif", ".jfi", ".gif", ".bmp", ".ico", ".jfif", ".jpe", ".pnz", ".dib", ".webp", ".exr", ".hdr", "dds",
+];
+
+pub const ZIP_FILES_EXTENSIONS: &[&str] = &[".zip"];
+
+pub const PDF_FILES_EXTENSIONS: &[&str] = &[".pdf"];
+
+pub const AUDIO_FILES_EXTENSIONS: &[&str] = &[
+    ".mp3", ".flac", ".wav", ".ogg", ".m4a", ".aac", ".aiff", ".pcm", ".aif", ".aiff", ".aifc", ".m3a", ".mp2", ".mp4a", ".mp2a", ".mpga", ".wave", ".weba",
+    ".wma",
+    /*".ac3",*/
+    /*".oga", https://github.com/Serial-ATA/lofty-rs/issues/47#issuecomment-1120414259*/
+];
+
+pub const VIDEO_FILES_EXTENSIONS: &[&str] = &[
+    ".mp4", ".mpv", ".flv", ".mp4a", ".webm", ".mpg", ".mp2", ".mpeg", ".m4p", ".m4v", ".avi", ".wmv", ".qt", ".mov", ".swf", ".mkv",
+];
 
 pub const LOOP_DURATION: u32 = 200; //ms
 
@@ -86,11 +116,9 @@ pub fn get_dynamic_image_from_raw_image(path: impl AsRef<Path> + std::fmt::Debug
         }
     };
 
-    let width = raw.width;
-    let height = raw.height;
     let source = ImageSource::Raw(raw);
 
-    let mut pipeline = match Pipeline::new_from_source(source, width, height, true) {
+    let mut pipeline = match Pipeline::new_from_source(source) {
         Ok(pipeline) => pipeline,
         Err(_e) => {
             return None;

czkawka_core/src/same_music.rs

@@ -7,13 +7,14 @@ use std::sync::atomic::{AtomicBool, AtomicUsize, Ordering};
 use std::sync::Arc;
 use std::thread::sleep;
 use std::time::{Duration, SystemTime};
-use std::{mem, thread};
+use std::{mem, panic, thread};
 
 use crossbeam_channel::Receiver;
 use lofty::{read_from_path, AudioFile, ItemKey};
 use rayon::prelude::*;
 use serde::{Deserialize, Serialize};
 
+use crate::common::AUDIO_FILES_EXTENSIONS;
 use crate::common::{open_cache_folder, Common, LOOP_DURATION};
 use crate::common_dir_traversal::{CheckingMethod, DirTraversalBuilder, DirTraversalResult, FileEntry, ProgressData};
 use crate::common_directory::Directories;
@@ -21,7 +22,6 @@ use crate::common_extensions::Extensions;
 use crate::common_items::ExcludedItems;
 use crate::common_messages::Messages;
 use crate::common_traits::*;
-use crate::similar_images::AUDIO_FILES_EXTENSIONS;
 
 #[derive(Eq, PartialEq, Clone, Debug)]
 pub enum DeleteMethod {
@@ -250,7 +250,7 @@ impl SameMusic {
 
     fn check_files(&mut self, stop_receiver: Option<&Receiver<()>>, progress_sender: Option<&futures::channel::mpsc::UnboundedSender<ProgressData>>) -> bool {
         if !self.allowed_extensions.using_custom_extensions() {
-            self.allowed_extensions.extend_allowed_extensions(&AUDIO_FILES_EXTENSIONS);
+            self.allowed_extensions.extend_allowed_extensions(AUDIO_FILES_EXTENSIONS);
         }
         let result = DirTraversalBuilder::new()
             .root_dirs(self.directories.included_directories.clone())
@@ -362,36 +362,43 @@ impl SameMusic {
                     return None;
                 }
 
-                let tagged_file = match read_from_path(&path, true) {
-                    Ok(t) => t,
-                    Err(_inspected) => {
-                        // println!("Failed to open {}", path);
-                        return Some(Some(music_entry));
+                let result = panic::catch_unwind(|| {
+                    match read_from_path(&path, true) {
+                        Ok(t) => Some(t),
+                        Err(_inspected) => {
+                            // println!("Failed to open {}", path);
+                            None
+                        }
+                    }
+                });
+
+                let tagged_file = match result {
+                    Ok(t) => match t {
+                        Some(r) => r,
+                        None => {
+                            return Some(Some(music_entry));
+                        }
+                    },
+                    Err(_) => {
+                        println!("File {} crashed during reading tags, please report bug", path);
+                        return Some(None);
                     }
                 };
 
                 let properties = tagged_file.properties();
 
-                let mut track_title;
-                let mut track_artist;
-                let mut year;
-                let mut length;
-                let mut genre;
+                let mut track_title = "".to_string();
+                let mut track_artist = "".to_string();
+                let mut year = "".to_string();
+                let mut genre = "".to_string();
 
                 let bitrate = properties.audio_bitrate().unwrap_or(0);
+                let mut length = properties.duration().as_millis().to_string();
 
-                let tag = match tagged_file.primary_tag() {
-                    Some(t) => t,
-                    None => {
-                        // println!("File {} don't have valid tag", path);
-                        return Some(Some(music_entry));
-                    }
-                };
-                {
+                if let Some(tag) = tagged_file.primary_tag() {
                     track_title = tag.get_string(&ItemKey::TrackTitle).unwrap_or("").to_string();
                     track_artist = tag.get_string(&ItemKey::TrackArtist).unwrap_or("").to_string();
                     year = tag.get_string(&ItemKey::Year).unwrap_or("").to_string();
-                    length = tag.get_string(&ItemKey::Length).unwrap_or("").to_string();
                     genre = tag.get_string(&ItemKey::Genre).unwrap_or("").to_string();
                 }
 
@@ -411,26 +418,23 @@ impl SameMusic {
                             year = tag_value.to_string();
                         }
                     }
-                    if length.is_empty() {
-                        if let Some(tag_value) = tag.get_string(&ItemKey::Length) {
-                            length = tag_value.to_string();
-                        }
-                    }
                     if genre.is_empty() {
                         if let Some(tag_value) = tag.get_string(&ItemKey::Genre) {
                             genre = tag_value.to_string();
                         }
                     }
+                    // println!("{:?}", tag.items());
                 }
 
-                // println!("{:?}", tag.items());
-
-                if let Ok(mut length_number) = length.parse::<u32>() {
-                    length_number /= 60;
+                if let Ok(old_length_number) = length.parse::<u32>() {
+                    let length_number = old_length_number / 60;
                     let minutes = length_number / 1000;
                     let seconds = (length_number % 1000) * 6 / 100;
                     if minutes != 0 || seconds != 0 {
                         length = format!("{}:{:02}", minutes, seconds);
+                    } else if old_length_number > 0 {
+                        // That means, that audio have length smaller that second, but length is properly read
+                        length = "0:01".to_string();
                     } else {
                         length = "".to_string();
                     }

czkawka_core/src/similar_images.rs

@@ -18,7 +18,7 @@ use image_hasher::{FilterType, HashAlg, HasherConfig};
 use rayon::prelude::*;
 use serde::{Deserialize, Serialize};
 
-use crate::common::{get_dynamic_image_from_raw_image, open_cache_folder, Common, LOOP_DURATION};
+use crate::common::{get_dynamic_image_from_raw_image, open_cache_folder, Common, IMAGE_RS_SIMILAR_IMAGES_EXTENSIONS, LOOP_DURATION, RAW_IMAGE_EXTENSIONS};
 use crate::common_directory::Directories;
 use crate::common_extensions::Extensions;
 use crate::common_items::ExcludedItems;
@@ -27,26 +27,6 @@ use crate::common_traits::{DebugPrint, PrintResults, SaveResults};
 use crate::flc;
 use crate::localizer_core::generate_translation_hashmap;
 
-pub const RAW_IMAGE_EXTENSIONS: [&str; 24] = [
-    ".mrw", ".arw", ".srf", ".sr2", ".mef", ".orf", ".srw", ".erf", ".kdc", ".kdc", ".dcs", ".rw2", ".raf", ".dcr", ".dng", ".pef", ".crw", ".iiq", ".3fr", ".nrw", ".nef", ".mos",
-    ".cr2", ".ari",
-];
-pub const IMAGE_RS_EXTENSIONS: [&str; 13] = [".jpg", ".jpeg", ".png", ".bmp", ".tiff", ".tif", ".tga", ".ff", ".jif", ".jfi", ".webp", ".gif", ".ico"];
-
-pub const IMAGE_RS_SIMILAR_IMAGES_EXTENSIONS: [&str; 9] = [
-    ".jpg", ".jpeg", ".png", ".tiff", ".tif", ".tga", ".ff", ".jif", ".jfi", //,".bmp",
-];
-pub const IMAGE_RS_BROKEN_FILES_EXTENSIONS: [&str; 10] = [
-    ".jpg", ".jpeg", ".png", ".tiff", ".tif", ".tga", ".ff", ".jif", ".jfi", ".gif", //,".bmp", ".ico"
-];
-pub const ZIP_FILES_EXTENSIONS: [&str; 1] = [".zip"];
-
-pub const AUDIO_FILES_EXTENSIONS: [&str; 8] = [".mp3", ".flac", ".wav", ".ogg", ".m4a", ".aac", ".aiff", ".pcm"];
-
-pub const VIDEO_FILES_EXTENSIONS: [&str; 16] = [
-    ".mp4", ".mpv", ".flv", ".mp4a", ".webm", ".mpg", ".mp2", ".mpeg", ".m4p", ".m4v", ".avi", ".wmv", ".qt", ".mov", ".swf", ".mkv",
-];
-
 pub const SIMILAR_VALUES: [[u32; 6]; 4] = [
     [0, 2, 5, 7, 14, 20],    // 8
     [2, 5, 15, 30, 40, 40],  // 16
@@ -297,8 +277,8 @@ impl SimilarImages {
         let mut folders_to_check: Vec<PathBuf> = Vec::with_capacity(1024 * 2); // This should be small enough too not see to big difference and big enough to store most of paths without needing to resize vector
 
         if !self.allowed_extensions.using_custom_extensions() {
-            self.allowed_extensions.extend_allowed_extensions(&IMAGE_RS_SIMILAR_IMAGES_EXTENSIONS);
-            self.allowed_extensions.extend_allowed_extensions(&RAW_IMAGE_EXTENSIONS);
+            self.allowed_extensions.extend_allowed_extensions(IMAGE_RS_SIMILAR_IMAGES_EXTENSIONS);
+            self.allowed_extensions.extend_allowed_extensions(RAW_IMAGE_EXTENSIONS);
         }
 
         // Add root folders for finding

czkawka_core/src/similar_videos.rs

@@ -17,6 +17,7 @@ use serde::{Deserialize, Serialize};
 use vid_dup_finder_lib::HashCreationErrorKind::DetermineVideo;
 use vid_dup_finder_lib::{NormalizedTolerance, VideoHash};
 
+use crate::common::VIDEO_FILES_EXTENSIONS;
 use crate::common::{open_cache_folder, Common, LOOP_DURATION};
 use crate::common_directory::Directories;
 use crate::common_extensions::Extensions;
@@ -25,7 +26,6 @@ use crate::common_messages::Messages;
 use crate::common_traits::{DebugPrint, PrintResults, SaveResults};
 use crate::flc;
 use crate::localizer_core::generate_translation_hashmap;
-use crate::similar_images::VIDEO_FILES_EXTENSIONS;
 
 pub const MAX_TOLERANCE: i32 = 20;
 
@@ -236,7 +236,7 @@ impl SimilarVideos {
         let mut folders_to_check: Vec<PathBuf> = Vec::with_capacity(1024 * 2); // This should be small enough too not see to big difference and big enough to store most of paths without needing to resize vector
 
         if !self.allowed_extensions.using_custom_extensions() {
-            self.allowed_extensions.extend_allowed_extensions(&VIDEO_FILES_EXTENSIONS);
+            self.allowed_extensions.extend_allowed_extensions(VIDEO_FILES_EXTENSIONS);
         }
 
         // Add root folders for finding

czkawka_gui/Cargo.toml

@@ -11,7 +11,7 @@ repository = "https://github.com/qarmin/czkawka"
 [dependencies]
 czkawka_core = { path = "../czkawka_core", version = "4.1.0"}
 gdk = "0.15.4"
-glib = "0.15.10"
+glib = "0.15.11"
 
 humansize = "1.1.1"
 chrono = "0.4.19"
@@ -26,10 +26,10 @@ futures = "0.3.21"
 directories-next = "2.0.0"
 
 # For opening files
-open = "2.1.1"
+open = "2.1.2"
 
 # To get image preview
-image = "0.24.1"
+image = "0.24.2"
 
 # To be able to use custom select
 regex = "1.5.5"
@@ -53,7 +53,7 @@ once_cell = "1.10.0"
 winapi = { version = "0.3.9", features = ["combaseapi", "objbase", "shobjidl_core", "windef", "winerror", "wtypesbase", "winuser"] }
 
 [dependencies.gtk]
-version = "0.15.4"
+version = "0.15.5"
 default-features = false # just in case
 features = ["v3_24_9"]
 

czkawka_gui/src/initialize_gui.rs

@@ -9,7 +9,8 @@ use gtk::prelude::*;
 use gtk::{CheckButton, Image, SelectionMode, TextView, TreeView};
 
 use crate::flg;
-use czkawka_core::similar_images::{IMAGE_RS_EXTENSIONS, RAW_IMAGE_EXTENSIONS, SIMILAR_VALUES};
+use czkawka_core::common::{IMAGE_RS_EXTENSIONS, RAW_IMAGE_EXTENSIONS};
+use czkawka_core::similar_images::SIMILAR_VALUES;
 use czkawka_core::similar_videos::MAX_TOLERANCE;
 
 use crate::create_tree_view::*;

czkawka_gui/src/saving_loading.rs

@@ -58,9 +58,9 @@ const DEFAULT_EXCLUDED_ITEMS: &str = "*/.git/*,*/node_modules/*,*/lost+found/*,*
 const DEFAULT_EXCLUDED_ITEMS: &str = "*\\.git\\*,*\\node_modules\\*,*\\lost+found\\*,*:\\windows\\*";
 
 #[cfg(target_family = "unix")]
-const DEFAULT_EXCLUDED_DIRECTORIES: [&str; 5] = ["/proc", "/dev", "/sys", "/run", "/snap"];
+const DEFAULT_EXCLUDED_DIRECTORIES: &[&str] = &["/proc", "/dev", "/sys", "/run", "/snap"];
 #[cfg(not(target_family = "unix"))]
-const DEFAULT_EXCLUDED_DIRECTORIES: [&str; 1] = ["C:\\Windows"];
+const DEFAULT_EXCLUDED_DIRECTORIES: &[&str] = &["C:\\Windows"];
 
 struct LoadSaveStruct {
     loaded_items: HashMap<String, Vec<String>>,

instructions/Compilation.md

@@ -9,7 +9,7 @@ FFmpeg is not included here, because is not needed to build because it is dynami
 
 | Program | Min  | What for                                                                      |
 |---------|------|-------------------------------------------------------------------------------|
-| Rust    | 1.57 | Czkawka, aims to support the latest available version of Rust on Ubuntu 20.04 |
+| Rust    | 1.60 | Czkawka, aims to support the latest available version of Rust on Ubuntu 20.04 |
 | GTK     | 3.24 | Only for the `GTK` backend                                                    |
 
 #### Debian / Ubuntu

instructions/Instruction.md

@@ -102,7 +102,8 @@ Windows - `C:\Users\Username\AppData\Local\Qarmin\Czkawka\cache`
   By default for all files grouped by same size are computed partial hash(hash from only of 2KB each file). Such hash is computed usually very fast, especially on SSD and fast multicore processors. But when scanning a hundred of thousands or millions of files with HDD or slow processor, usually this step can take much time. In settings exists option `Use prehash cache` which enables caching such things. It is disabled by default because can increase time of loading/saving cache, with big number of entries.
 - **Permanent store of cache entries**  
   After each scan, entries in cache are validated and outdated ones(which points at non-existent files) are removed. This may be problematic when scanning external drivers(like pendrives, disks etc.) and later unplugging and plugging them again. In settings exists option `Delete outdated cache entries automatically` which automatically clear this, but this can be disabled. Disabling such option may create big cache files, so button `Remove outdated results` will do it manually.
-
+- **Partial scanning**
+  If you know that you can't scan all files at once, you can still try to scan all files and during scan just stop it, so already calculated hashes/data will be saved to cache and will speedup later scans.
 
 # Tools
 
@@ -272,3 +273,7 @@ It works in this way:
 - Returns all file extensions that are connected to this mime type e.g. `rar,7z,zip,p7`
 - Basing on file extension, adds more elements to list from above(needed because some files e.g. `exe` and `dll` begins with similar/same bytes)
 - If current file extensions is inside list then probably have proper extension, if is not inside, then is shown as file with invalid extension
+
+In `Proper Extension` column, inside parentheses is visible extension guessed by Infer library, and outside there are extensions which have same mime type as guessed one. 
+![ABC](https://user-images.githubusercontent.com/41945903/167214811-7d811829-6dba-4da0-9788-9e2f780e7279.png)
+