Skip to content

Commit

Permalink
Fix Python 3.8 multiprocessing issue(?); note: QProgressBar disappear…
Browse files Browse the repository at this point in the history
…s using 'Fusion' on Windows when opening a file explorer
  • Loading branch information
ewancook committed May 13, 2020
1 parent e72f134 commit c86637a
Show file tree
Hide file tree
Showing 3 changed files with 16 additions and 17 deletions.
15 changes: 7 additions & 8 deletions transcriber/collator/model.py
Original file line number Diff line number Diff line change
Expand Up @@ -21,15 +21,14 @@ def __init__(self):
self.state = False

@QtCore.pyqtSlot(str, list)
def collate(self, save_file, filenames):
def collate(self, collated_file, filenames):
self.collation_started.emit()
with open(save_file, "w") as collated_file:
self.process = Process(
target=utils.collate, args=(collated_file, filenames,)
)
self.process.start()
self.state = True
self.process.join()
self.process = Process(
target=utils.collate, args=(collated_file, filenames,)
)
self.process.start()
self.state = True
self.process.join()
if self.state:
logging.info(
f"Collation of {len(filenames)} file(s) was successful"
Expand Down
8 changes: 4 additions & 4 deletions transcriber/collator/test_collator.py
Original file line number Diff line number Diff line change
Expand Up @@ -40,14 +40,14 @@ def test_collator_collate(
mock_join.assert_called_once()

@mock.patch("transcriber.collator.utils.collate_files")
def test_collate(self, mock_collate):
save_file = mock.Mock()
@mock.patch("builtins.open", new_callable=mock.mock_open)
def test_collate(self, mock_file, mock_collate):
save_file = "this/is/the/save/file.csv"
filenames = ["this/is/a/test.csv"]
utils.collate(save_file, filenames)
mock_collate.assert_called_with(
save_file, [transcribed_filename(f) for f in filenames]
mock_file(), [transcribed_filename(f) for f in filenames]
)
save_file.close.assert_called_once()

def test_collate_files(self):
num_files = 3
Expand Down
10 changes: 5 additions & 5 deletions transcriber/collator/utils.py
Original file line number Diff line number Diff line change
Expand Up @@ -3,11 +3,11 @@
from transcriber.converter.dbfworker import utils


def collate(save_file, filenames):
collate_files(
save_file, [utils.transcribed_filename(f) for f in filenames],
)
save_file.close() # parent won't close this, so we have to
def collate(collated_file, filenames):
with open(collated_file, "w") as save_file:
collate_files(
save_file, [utils.transcribed_filename(f) for f in filenames],
)


def collate_files(collated_file, filenames):
Expand Down

8 comments on commit c86637a

@cshoopman
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is there any way I could automate this to where I provide the Float file I want and the Tagname File I want in some sort of script and it spits out the result with the naming of my choosing? I wouldn't need any options other than selecting all Tagnames.

@ewancook
Copy link
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is there any way I could automate this to where I provide the Float file I want and the Tagname File I want in some sort of script and it spits out the result with the naming of my choosing? I wouldn't need any options other than selecting all Tagnames.

Hey, there is no way to do this in v.015 (current version) - it's planned for the next release, but that's quite a long way off...

I saw your comment and spent this evening writing a tool for this purpose! Please see #4. It also has additional features:

  • Row Averaging
  • Precision Adjustment

These are both highly recommended to reduce file sizes!

Let me know how it goes. Cheers.

@cshoopman
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you so much! I really appreciate it. I'll test it out and let you know how it works for me.

@cshoopman
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Works great, does everything I need it to do. Trying to convert the daily DAT files into CSV files and then I can further process those. Thanks from a fellow (I think) ChemE.

@cshoopman
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

On a few files, I get the following error:
Transcription failed: seek out of range (2020 10 20 0000 (Float).DAT)
It doesn't matter if I do it via command line or through the GUI. What does this mean and is there a way to work around it?

@ewancook
Copy link
Owner Author

@ewancook ewancook commented on c86637a Jan 4, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

On a few files, I get the following error:
Transcription failed: seek out of range (2020 10 20 0000 (Float).DAT)
It doesn't matter if I do it via command line or through the GUI. What does this mean and is there a way to work around it?

Assuming you're using the correct tag file, it looks like you've run into issue #3, and that trends have been added/removed midway through the file. Unfortunately, I wrote the entire thing before realising that trends could change midway through files - in hundreds of days of data I had never seen it happen.

Fixing this involves almost a complete converter rewrite and changes a lot of the GPU logic as well: it'll need to accept pairs of Float/Tag files (i.e. a folder), and will only show tags that are present in all files, for example. A lot of it is done, but realistically I have no more time to work on it until at least the end of January, and even then I won't have much...

In the meantime, I'd suggest that you:

  • Convert your files to a CSV using FactoryTalk SE File Viewer (I think the link's correct, but you'll need to log in).
  • Read Transcription when trends change (v0.15) #3 and use FactoryTalk SE File Viewer to see if trends do change.
  • Reformat the CSV using a very early version of transcriber, written just before the DBF conversion was included (original purpose was to reformat CSV files as above!). I can't guarantee this will work, but it's probably your best bet for now... Having had a quick check, I can almost guarantee it won't work!

Sorry, but it looks like your best bet for now is using it where you can, because it looks like that might be the only way until I fix #3.

Are you aware of any projects that your software team might be working on that would mean trends are being added/removed on a regular basis?

@cshoopman
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

On a few files, I get the following error:
Transcription failed: seek out of range (2020 10 20 0000 (Float).DAT)
It doesn't matter if I do it via command line or through the GUI. What does this mean and is there a way to work around it?

Assuming you're using the correct tag file, it looks like you've run into issue #3, and that trends have been added/removed midway through the file. Unfortunately, I wrote the entire thing before realising that trends could change midway through files - in hundreds of days of data I had never seen it happen.

Fixing this involves almost a complete converter rewrite and changes a lot of the GPU logic as well: it'll need to accept pairs of Float/Tag files (i.e. a folder), and will only show tags that are present in all files, for example. A lot of it is done, but realistically I have no more time to work on it until at least the end of January, and even then I won't have much...

In the meantime, I'd suggest that you:

  • Convert your files to a CSV using FactoryTalk SE File Viewer (I think the link's correct, but you'll need to log in).
  • Read Transcription when trends change (v0.15) #3 and use FactoryTalk SE File Viewer to see if trends do change.
  • Reformat the CSV using a very early version of transcriber, written just before the DBF conversion was included (original purpose was to reformat CSV files as above!). I can't guarantee this will work, but it's probably your best bet for now... Having had a quick check, I can almost guarantee it won't work!

Sorry, but it looks like your best bet for now is using it where you can, because it looks like that might be the only way until I fix #3.

Are you aware of any projects that your software team might be working on that would mean trends are being added/removed on a regular basis?

That makes sense as to why it doesn't work.

If I find the offending tag(s) and don't select it will that make it work? I remember for one of the days I had to de-select a couple of trends (it told me exactly which ones) because they weren't in the Float.

I think this is the correct link for FactoryTalk SE File Viewer.

This issue almost definitely occurred because I was adding a new tag to our trend or removing an old one that was no longer in use. We brought a dormant plant back and some of the things that were there are no longer there. Some new things were added and I wanted to get them logging. It shouldn't be an overly common occurrence going forward. I am debating on making all of the trending log to a database instead of the DAT files.

@ewancook
Copy link
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If I find the offending tag(s) and don't select it will that make it work?

Sadly not. Transcriber reads rows in sets of X (total tags) at a time as an optimisation. So if you wanted the first and third tag in a set of a hundred tags (X), it would read the first tag, jump to the third tag (and read), then jump to first tag in the next set of X. This reduces the time taken when you only select a few tags - you'll have maybe 30,000 calls to read() and seek() instead of upwards of sometimes 3,000,000+ (if you read every row each time, regardless of how many tags were selected). The problem here is that X is determined up front (i.e. before the tags change midway through), and while I've fixed most of the converter code, I haven't thought of an optimal way to "transpose" it into columns yet - some columns obviously stop or start midway through.

I remember for one of the days I had to de-select a couple of trends (it told me exactly which ones) because they weren't in the Float.

So this was something I added to allow use of the same tag file for ease, but it unfortunately doesn't work when trends are added/removed midway through float files.

It shouldn't be an overly common occurrence going forward. I am debating on making all of the trending log to a database instead of the DAT files.

Yeah I think the whole DAT environment is regarded as a legacy system by FactoryTalk? Probably best to migrate if possible (if exporting to CSV is easy)!

Thanks for the update!

Please sign in to comment.