Home > database >  processing file
processing file

Time:10-13

After thinking through the solutions and analyzing all answers, i have deleted my post

CodePudding user response:

The point of a BackgroundWorker is to push things off the UI thread. You do that, but then in Bw_DoWork you immediately push things back the UI thread, with this.Dispatcher.Invoke, which blocks the worker thread until the work now running on the UI thread completes.

Basically: remove that this.Dispatcher.Invoke call, and just run the code on the worker. If you need to touch the UI, then you'll need to be on the UI thread for those bits, but only those buts.

Likewise, I doubt that mutliocr should be using dispatch invoke, and it certainly shouldn't change the UI thread to being a background thread (Thread.CurrentThread.IsBackground = true;).

CodePudding user response:

Your code is very very smelly and ugly. You seem to follow absolutely no naming convention at all. Locals and class members are camelCase and PascalCase and some use no casing at all and others use underscores. You should really review you code carefully with many aspects in mind and clean it up. There are some redundancies too.

A really bad habbit of yours is the excessive use of the Dispatcher. For example, you create a background thread and post the complete! work of this thread back to the Dispatcher/UI thread. A huge waste of resources and elimination of any multithreading benefit.
You don't want to put all your work on the Dispatcher. You want to offload CPU intensive eork to a background thread. You want to make use of asynchronous APIs where possible. Because you want to keep the UI responsive. Dispatcher means UI thread.

Some highly important points of interest

  • GrantAccess implementation is a severe security violation of user rights. Don't modify access rules. Rather filter and ignore resources where the current user is not authorized for access. What is especially critical, is that you never revert the access rights to the original state.
  • As a general rule: don't execute IO related code (e.g., database, HTTP streams, file IO) on a background thread. Use async APIs instead. Use threading only for CPU bound work (e.g. computations, conversions).
  • File has an async API you should always use. For example File.OpenRead returns a FileStream which exposes a FileStream.ReadAsnyc member. if you need more convenience when handling files like read line by line, then wrap the FileStream into a StreamReader/StreamWriter and use members like StreamReader.ReadLineAsync
  • To avoid Dispatcher calls, pass the required UI values to the concurent methods as argument. Better use data binding (which won't eliminate the cross threading issues writing, but would make your code more readable and eliminates Dispatcher invocation to read the values). Take alook at the refactored File_process method below. It shows how to pass UI values as argument to the converttiffpdfreducer method which is then executed on a background thread.
  • Consider to add cancellation support to to the longrunning converttiffpdfreducer()
  • Avoid calling ToArray or ToList on an IEnumerable. Those methods are finalizers that immediately execute the originally deferred LINQ queries.
  • Don't call ToArray or ToList on the result of EnumerateFiles and its likes. Those methods are used to improve performance as they return the filesystem objects item by item. This is especially important if you recursively iterate over the complete filesystem structure. If the filesystem tree is very deep and broad, calling ToArray will force the itereation to complete and then to return all results at once. ToArray on EnumerateFiles is like using GetFiles. You should review your complete code and refactor it properly. You always use EnumerateXYZ wrong!

"The EnumerateFiles and GetFiles methods differ as follows: When you use EnumerateFiles, you can start enumerating the collection of names before the whole collection is returned. When you use GetFiles, you must wait for the whole array of names to be returned before you can access the array. Therefore, when you are working with many files and directories, EnumerateFiles can be more efficient."

  • Use data binding instead of directly accessing UI elements. This allows you to read the properties on the background thread without using the Dispatcher.
  • Never execute long running operation from the constructor
  • Never call async code from the constructor
  • Always keep object instantiation cheep and fast and without hidden performance/resource costs
  • Never catch Exception. Always catch a specialized exception type.
  • Don't use empty catch blocks. Either handle the exception if you can or laet it crash your application to give you a chance to fix bugs. When you swallow exceptions bugs will silently creep into your application. You will have a really hard time to discover them. Logging exceptions is not considered handling - rethrow in such case.
  • You don't have to close a resource explicitly if you declare the resource using the using statement. The implicit call to Dispose once the instruction pointer leaves the using scope will close the resource automatically.

Implementing all the suggestion will significantly speed up your application.

I have refactored only some of your code to show how to properly use async APIs and Task.Run instead of the BackgroundWorker. I have removed every Dispatcher invocation. Instead of direct access to UI elements in order to read their values from the background thread, I have extracted those values before invoking the concurent method and passed those prefetched values as method arguments. If you would use data binding you could read the property values directly and therefore ommit the method parameters.

The MainWindow should be shown manually from App.xaml.cs to allow asynchronous and longrunning initialization of the instance. For this pattern let the class that requires such initialization implement a public InitializeAsync method that can be awaited from the caller's context. Alternatively use Lazy<T> to defer initialization when required e.g., when initialization is depending on explicit access to members.

Although the refactored code will significantly improve the applicatoin's performance, you will have do to do some important refactoring yourself (following the pattern of the already refactored code sections).
Take a look at

  • InitializeAsync and WriteLnAsync to learn how to use the async file IO API.
  • converttiffpdfreducer to learn how to use the EnumerateFiles and EnumerateDirectories methods properly in order to significantly improve the performance.
  • mutliocr, merge and converttiffpdfreducer to learn how pass UI element values as argument in order to avoid Dispatcher invocations.
  • start_btn_Click and converttiffpdfreducer to learn how to implemnent cancellation and to guard your API against calls during an uninitialized state

App.xaml

<Application Startup="Application_Startup">

</Application>

App.xaml.cs

class App : Application
{
  private async void Application_Startup(object sender, StartupEventArgs e)
  {
    var mainWindow = new MainWindow();

    // Because InitializeAsync depends on UI elements,
    // we have to wait until the Ui is loaded.
    mainWindow.Loaded  = OnMainWindowLoaded;

    // Either call Show() before initialization or after.
    // If before, ensure access to uninitialized members and resources is denied
    // e.g. by querying the MainWindow.IsInitialized property in public members and event handlers.
    mainWindow.Show();        
  }

  private async void OnMainWindowLoaded(object sender, EventArgs args) 
    => await mainWindow.InitializeAsync();
}

MainWindow.xaml.cs

public partial class MainWindow : Window
{
  public bool IsInitialized { get; private set; }
  private bool IsBusy { get; set; }
  private CancellationTokenSource CancellationTokenSource { get; set; }

  public MainWindow()
  {
    InitializeComponent();
    CancellationTokenSource = new CancellationTokenSource();
  }

  // Execute blocking initialization routines asynchronously
  public async Task InitializeAsync()
  {
    if (IsInitialized)
    {
      return;
    }

    // Will execute the intesive CPU bound work on a background thread.
    await File_process(Cancellationtoken.None);

    string configpath = Path.Combine(AppDomain.CurrentDomain.BaseDirectory, @"path.txt");
    // Use async API to read/write from/to files and other IO resources
    using (FileStream configfile = File.OpenRead(configpath))
    {
      using (var fileReader = new StreamReader(configfile))
      {
        var configFileContent = new List<string>();
        while (!fileReader.EndOfStream)
        {
          string lineOfFile = await fileReader.ReadLineAsync();
          configFileContent.Add(lineOfFile);
        }

        if (configFileContent.Any())
        {
          ip.Text = configFileContent[0];
          GrantAccess(configFileContent[0]);

          op.Text = configFileContent[1];
          GrantAccess(configFileContent[1]);

          ex_tb.Text = configFileContent[2];
          GrantAccess(configFileContent[2]);

          Protb.Text = configFileContent[3];
          GrantAccess(configFileContent[3]);
        }
      }
    }

    cbPDFConform.Items.Clear();
    for (int i = 0; i < Enum.GetNames(typeof(PdfConformance)).Length - 1; i  )
    {
      ComboBoxItem cbi = new ComboBoxItem();
      cbi.Content = Enum.GetName(typeof(PdfConformance), (PdfConformance)i);
      PdfConformance test = (PdfConformance)i;
      cbi.Tag = (PdfConformance)i;
      cbPDFConform.Items.Add(cbi);
    }

    cbPDFConform.SelectedIndex = 0;
    cbProcessorCount.Items.Clear();
    for (int i = 1; i <= Environment.ProcessorCount; i  )
    {
      cbProcessorCount.Items.Add(i.ToString());
      if (Environment.ProcessorCount / 2 == i) { cbProcessorCount.SelectedIndex = i - 1; }
    }

    LicenseManager oLicenseManager = new LicenseManager();
    oLicenseManager.RegisterKEY("");

    threadSyn = SynchronizationContext.Current;

    IsInitialiezd = true;
  }

  public async Task File_process(CancellationToken cancellationToken)
  {
    // Read UI values to avoid Dispatcher calls from the background thread
    string ipText = ip.Text;
    string protbText = Protb.Text;
    string opText = op.Text;

    // Execute the intesive CPU bound work on a background thread.
    await Task.Run(() => converttiffpdfreducer(ipText, protbText, opText, cancellationToken), cancellationToken);
  }

private async Task DoWorkAsync(CancellationToken cancellationToken)
{
  IsBusy = true;

  using (var sw = new StreamWriter(Path.Combine(AppDomain.CurrentDomain.BaseDirectory, @"path.txt"), false))
  {
    await sw.WriteLineAsync(ip.Text);
    await sw.WriteLineAsync(op.Text);
    await sw.WriteLineAsync(ex_tb.Text);
    await sw.WriteLineAsync(Protb.Text);
  }

  try
  {
    cancellationToken.ThrowIfCancellationRequested();

    var watch1 = new System.Diagnostics.Stopwatch();
    watch1.Start();

    // Consider to add cancellation support to File_process 
    await File_process(cancellationToken);

    watch1.Stop();

    TimeSpan ts1 = watch1.Elapsed;
    ts1.ToString("mm\\:ss");

    if (MergeChk.IsChecked == false)
    {
      value = "OCRed";
    }

    await WriteLnAsync("All documents have been successfully "   value   " "   ts1   " "   DateTime.Now   " "   Environment.UserName, cancellationToken);

    IsBusy = false;
  }
  catch (OperationCanceledException)
  {
    IsBusy = false;
    throw;
  }
}

  private async Task WriteLnAsync(string text, CancellationToken cancellationToken)
  {
    logtb.Text  = text   Environment.NewLine;

    log_list.Add(text);
    log_cap = text   Environment.NewLine   log_cap;
    using (var sw = new StreamWriter(Path.Combine(AppDomain.CurrentDomain.BaseDirectory, @"FileProcessing_log.txt"), false))
    {
      foreach (string l in log_list)
      {
        cancellationToken.ThrowIfCancellationRequested();
        await sw.WriteLineAsync(l);
      }
    }
  }

  private async void start_btn_Click(object sender, RoutedEventArgs e)
  {
    if (!IsInitialized)
    {
      return;
    }

    if (IsBusy)
    {
      // Cancel the longrunning operation.
      this.CancellationTokenSource.Cancel();
    }
    start_btn.Content = "Start";

    try
    {
      await DoWorkAsync(CancellationTokenSource.Token);
    }
    catch (OperationCanceledException)
    {
      CancellationTokenSource?.Dispose();
      CancellationTokenSource = new CancellationTokenSource();
    }
  }

  private void converttiffpdfreducer(
    string ipText, 
    string protbText, 
    string opText, 
    CancellationToken cancellationToken)
  {
    string[] dir = null;
    string box = string.Empty;
    string box1 = string.Empty;
    string[] gg = null;


    PdfConformance optPDFConform = PdfConformance.Unknown;

    foreach (var directoryPath in Directory.EnumerateDirectories(ipText, "*.*", SearchOption.AllDirectories).Where(l => l.Length != 0))
    {
      cancellationToken.ThrowIfCancellationRequested();

      foreach (var filePath in Directory.EnumerateFiles(directoryPath, "*.*", SearchOption.AllDirectories))
      {
        cancellationToken.ThrowIfCancellationRequested();

        string getext = Path.GetExtension(filePath);
        string fd = Path.GetDirectoryName(filePath);
        string op_path = fd.Replace(ipText, protbText);
        string getextension = Path.GetExtension(filePath);
        string dict = Path.Combine(AppDomain.CurrentDomain.BaseDirectory, "Redist", "OCR");
        string outputPath = fd.Replace(ipText, protbText);
        string FNAME = Path.GetFileNameWithoutExtension(filePath);
        string fn = Path.GetDirectoryName(filePath).Replace(ipText, protbText);
        string filen = Path.Combine(outputPath, fn, FNAME   ".pdf");
        string savefile = Path.Combine(op_path, filen);

        box = Path.GetDirectoryName(filePath);
        box1 = Path.GetDirectoryName(box);
        using (GdPictureDocumentConverter oConverter = new GdPictureDocumentConverter())
        {
          GdPictureStatus status = new GdPictureStatus();
          if (Path.GetExtension(filePath).ToUpper() == ".PDF")
          {
            status = oConverter.LoadFromFile(filePath, GdPicture14.DocumentFormat.DocumentFormatPDF);
          }

          else if (Path.GetExtension(filePath).ToUpper() == ".TIF" || Path.GetExtension(filePath).ToUpper() == ".TIFF")

          {
            status = oConverter.LoadFromFile(filePath, GdPicture14.DocumentFormat.DocumentFormatTIFF);
          }
          else if (Path.GetExtension(filePath).ToUpper() == ".JPG")
          {
            status = oConverter.LoadFromFile(filePath, GdPicture14.DocumentFormat.DocumentFormatJPEG);
          }

          if (status == GdPictureStatus.OK)
          {
            if (!Directory.Exists(op_path))
            {
              Directory.CreateDirectory(op_path);
            }
            GrantAccess(op_path);
            optPDFConform = (PdfConformance)((ComboBoxItem)cbPDFConform.SelectedItem).Tag;
            status = oConverter.SaveAsPDF(savefile, optPDFConform);
            if (status == GdPictureStatus.OK)
            { }
            else
            { }
          }
          else
          { }
        }
      }
    }

    string BOXX = box.Replace(ipText, protbText);

    // TODO::Refactor 'merge' and replace 'ToArray' with 'foreach'
    string[] Arr = Directory.EnumerateFiles(BOXX, "*.pdf", SearchOption.AllDirectories).ToArray();
    if (MergeChk.IsChecked == true)
    { merge(Arr, protbText); }
    else if (MergeChk.IsChecked == false)
    {
      mutliocr(Arr);
    }
  }

  private string[] merge(string[] arr, string protbText, string opText)
  {
    string box = string.Empty;
    string box1 = string.Empty; string[] gg = null;
    System.Windows.Threading.Dispatcher.CurrentDispatcher.Invoke((Action)(() =>
    {
      box = Path.GetDirectoryName(arr[0]);
      box1 = Path.GetDirectoryName(box);
      string dirName = Directory.GetParent(arr[0]).FullName;
      string BOXFILES = Path.GetDirectoryName(dirName);
      string folder = Directory.GetParent(arr[0]).FullName.Replace(protbText, opText);

      string ocrfolder = (new FileInfo(arr[0]).Directory.FullName).Replace(protbText, opText);
      string fn = Directory.GetParent(arr[0]).Name;
      string filen = Path.Combine(ocrfolder, folder, fn   ".pdf");
      if (!Directory.Exists(ocrfolder))
      {
        Directory.CreateDirectory(ocrfolder);
      }
      GrantAccess(ocrfolder);
      using (GdPicturePDF oGdPicturePDF = new GdPicturePDF())
      {
        GdPictureStatus status = oGdPicturePDF.MergeDocuments(ref arr, filen);

        if (status == GdPictureStatus.OK)
        { }
        else
        { }
        oGdPicturePDF.Dispose();
      }
      Directory.Delete(box, true);
      string BOXX = box.Replace(protbText, opText);//op            
      string[] files = Directory.EnumerateFiles(BOXX, "*.pdf", SearchOption.AllDirectories).ToArray();
      if (MergeChk.IsChecked == true)
      { mutliocr(files, protbText, opText); }
    }));
    return gg;
  }

  private string[] mutliocr(string[] arr, string protbText, string opText)
  {
    string box = string.Empty;
    string box1 = string.Empty;
    try
    {
      string filepath = string.Empty;
      if (MergeChk.IsChecked == true)
      { filepath = opText; }
      else if (MergeChk.IsChecked == false)
      { filepath = protbText; }
      System.Windows.Threading.Dispatcher.CurrentDispatcher.Invoke((Action)(() =>
      {
        Thread.CurrentThread.IsBackground = true;
        var watch2 = new System.Diagnostics.Stopwatch();
        watch2.Start();
        string[] getfilearray = arr;
        for (int f = 0; f < getfilearray.Length; f  )
        {
          string dirName = Directory.GetParent(getfilearray[f]).FullName;
          string folder = Directory.GetParent(getfilearray[f]).FullName;

          box = Path.GetDirectoryName(getfilearray[f]);
          box1 = Path.GetDirectoryName(box);
          string getextension = Path.GetExtension(getfilearray[f]);
          string[] newF = Directory.EnumerateFiles(dirName, "*.*", SearchOption.AllDirectories).ToArray();
          string FN = Directory.GetParent(getfilearray[f]).Name;
          string ocrfolder = (new FileInfo(getfilearray[f]).Directory.FullName);
          string filen = Path.Combine(ocrfolder, folder, FN   "-ocr"   getextension);
          string dict = Path.Combine(AppDomain.CurrentDomain.BaseDirectory, "Redist", "OCR");
          if (!Directory.Exists(ocrfolder))
          {
            Directory.CreateDirectory(ocrfolder);
          }
          GrantAccess(ocrfolder);
          GdPicturePDF oGdPicturePDF = new GdPicturePDF();
          oGdPicturePDF.OcrPagesDone  = OcrPagesDone;
          void OcrPagesDone(GdPictureStatus status1)
          {
            if (oGdPicturePDF.SaveToFile(filen) == GdPictureStatus.OK)
            { }
            else
              MessageBox.Show("PDF: The OCR-ed file has failed to save. Status: "   oGdPicturePDF.GetStat().ToString());
          }
          GdPictureStatus status = GdPictureStatus.OK;
          if (oGdPicturePDF.LoadFromFile(getfilearray[f], false) == GdPictureStatus.OK)
            if (status == GdPictureStatus.OK)
            {
              if (oGdPicturePDF.OcrPages_4("*", 0, "eng", dict, "", 300, OCRMode.FavorSpeed, 1, true) == GdPictureStatus.OK)
                if (status == GdPictureStatus.OK)
                { }
                else
                { MessageBox.Show("PDF: The OCR process has failed. Status: "   status.ToString()); }
            }
            else
            { MessageBox.Show("PDF: The PDF file has failed to load. Status: "   status.ToString()); }

          oGdPicturePDF.Dispose();
          GrantAccess(getfilearray[f]);
          File.Delete(getfilearray[f]);
          watch2.Stop();
          TimeSpan ts2 = watch2.Elapsed;
          ts2.ToString("mm\\:ss");
          WriteLn(" OCR pages "   filen.Replace(opText, "")   " "   ts2   " "   DateTime.Now);
        }
        if (MergeChk.IsChecked == true)
        {
          foreach (string str in Directory.EnumerateFiles(opText, "*.pdf", SearchOption.AllDirectories).ToArray())
          {
            if (Path.GetFileNameWithoutExtension(str).EndsWith("-ocr"))
              File.Move(str, Path.Combine(Path.GetDirectoryName(str), Path.GetFileNameWithoutExtension(str).Substring(0, Path.GetFileNameWithoutExtension(str).Length - 4)   ".pdf"));
          }
        }
        if (MergeChk.IsChecked == false)
        {
          FileSystem.MoveDirectory(protbText, opText, UIOption.AllDialogs);
          Directory.CreateDirectory(protbText);
          string FF = string.Empty;
          foreach (string str in Directory.EnumerateFiles(opText, "*.pdf", SearchOption.AllDirectories))
          {

            if (Path.GetFileNameWithoutExtension(str).EndsWith("-ocr"))
              File.Move(str, Path.Combine(Path.GetDirectoryName(str), Path.GetFileNameWithoutExtension(str).Substring(0, Path.GetFileNameWithoutExtension(str).Length - 4)   ".pdf"));
          }
        }

      }));
    }
    catch (Exception mul)
    {
    }
    return arr;
  }
}
  • Related