-
Notifications
You must be signed in to change notification settings - Fork 46
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Parallel py3 #43
base: master
Are you sure you want to change the base?
Parallel py3 #43
Conversation
…ger throws an error
Thanks very much for this great contribution @mxndrwgrdnr! We'll review this soon, sorry for the delay. There is an existing earlier open PR that was reveiewed, which we kept open for a bit in the case anyone from the community wanted to review, that we'll merge in first. And then we can help reconcile with this PR. |
synthpop/recipes/starter2.py
Outdated
self.p_pums_cols = ('serialno', 'PUMA00', 'PUMA10', 'RELP', 'AGEP', | ||
'ESR', 'RAC1P', 'HISP', 'SEX') | ||
self.p_pums_cols = ('serialno', 'SPORDER', 'PUMA00', 'PUMA10', 'RELP', 'AGEP', | ||
'ESR', 'SCHL', 'SCH', 'JWTR', 'PERNP', 'WKHP', 'RAC1P', 'HISP', 'SEX') |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Why are you adding these variables if they are not used in the synthesis?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
They will be.
Awesome, thanks @mxndrwgrdnr! I'll check it out today |
thanks !! @mxndrwgrdnr |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Looks good to me @mxndrwgrdnr
synthpop/ipu/ipu.py
Outdated
@@ -259,9 +260,12 @@ def household_weights( | |||
iterations += 1 | |||
|
|||
if iterations > max_iterations: | |||
raise RuntimeError( | |||
warnings.warn( |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
In a conv with @janowicz we discussed that it would be better if this ignoring of errors was made optional. Could we make an ignore
parameter, where, if set to True, would raise warnings instead of errors?
@@ -169,7 +169,7 @@ def test_household_weights( | |||
def test_household_weights_max_iter( | |||
household_freqs, person_freqs, household_constraints, | |||
person_constraints): | |||
with pytest.raises(RuntimeError): | |||
with pytest.warns(UserWarning): |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Idem comment to ipu.py line263.
I tested the branch with a local environment in my computer and two Future Warnings keep coming up:
2 - synthpop\census_helpers.py: Line 180:
|
@janowicz the second FutureWarning that @cvanoli mentions isn't a big deal and should resolve itself in the future. The first FutureWarning, however, is happening during IPU when household weights that are very nearly zero are not close enough to zero to be caught by |
…rs flag to trigger the UserWarning behavior instead
added python 3.7
…parallel_py3 acs year missing
max_iterations
is reached, it generates a warning.synthesize_all_in_parallel()
to synthesizer.py for processing sub-county geographies in parallel.Could use an extra set of eyes at L264-265 in synthesizer.py. These two lines account for the fact that when processing geographies in parallel,
draw.draw_households()
cannot rely on thehh_index_start
argument to set thehousehold_id
either as the index of the synthetic households table or thehh_id
field of the synthetic persons table. Thus we have to adjust the households index and personshh_id
field AFTER all the tables have been generated, which is what's hapenning at L264-265.