Estimating Suppressed QCEW Data

[matthewlchambers]

Trying to adapt the bls_clean_allocation_fba_w_sec() function from BLS_QCEW.py to the new format, but I'm having a difficult time figuring out how this function is estimating the suppressed cells. Could someone briefly explain what's supposed to be happening in this function? Thanks!

[matthewlchambers]

I think this function accomplishes the desired goal. @catherinebirney or @bl-young , could you look it over and tell me what you think? Or if I've just misunderstood what we're trying to accomplish?

flowsa/flowsa/data_source_scripts/temp_data_source_functions.py

Lines 54 to 122 in f182edd

	def estimate_suppressed_qcew(fba: FlowByActivity) -> FlowByActivity:
	indexed = (
	fba
	.assign(n2=fba.ActivityProducedBy.str.slice(stop=2),
	n3=fba.ActivityProducedBy.str.slice(stop=3),
	n4=fba.ActivityProducedBy.str.slice(stop=4),
	n5=fba.ActivityProducedBy.str.slice(stop=5),
	n6=fba.ActivityProducedBy.str.slice(stop=6),
	location=fba.Location,
	category=fba.FlowName)
	.replace({'FlowAmount': {0: np.nan},
	'ActivityProducedBy': {'31-33': '3X',
	'44-45': '4X',
	'48-49': '4Y'},
	'n2': {'31': '3X', '32': '3X', '33': '3X',
	'44': '4X', '45': '4X',
	'48': '4Y', '49': '4Y'}})
	.set_index(['n2', 'n3', 'n4', 'n5', 'n6', 'location', 'category'],
	verify_integrity=True)
	)
	def fill_suppressed(
	flows: pd.Series,
	level: int,
	full_naics: pd.Series
	) -> pd.Series:
	parent = flows[full_naics.str.len() == level]
	children = flows[full_naics.str.len() == level + 1]
	null_children = children[children.isna()]
	if null_children.empty or parent.empty:
	return flows
	else:
	value = max((parent[0] - children.sum()) / null_children.size, 0)
	return flows.fillna(pd.Series(value, index=null_children.index))
	unsuppressed = (
	indexed
	.assign(
	FlowAmount=lambda x: (
	x.groupby(level=['n2',
	'location', 'category'])['FlowAmount']
	.transform(fill_suppressed, 2, x.ActivityProducedBy)))
	.assign(
	FlowAmount=lambda x: (
	x.groupby(level=['n2', 'n3',
	'location', 'category'])['FlowAmount']
	.transform(fill_suppressed, 3, x.ActivityProducedBy)))
	.assign(
	FlowAmount=lambda x: (
	x.groupby(level=['n2', 'n3', 'n4',
	'location', 'category'])['FlowAmount']
	.transform(fill_suppressed, 4, x.ActivityProducedBy)))
	.assign(
	FlowAmount=lambda x: (
	x.groupby(level=['n2', 'n3', 'n4', 'n5',
	'location', 'category'])['FlowAmount']
	.transform(fill_suppressed, 5, x.ActivityProducedBy)))
	.fillna({'FlowAmount': 0})
	.reset_index(drop=True)
	)
	aggregated = (
	unsuppressed
	.assign(FlowName='Number of employees')
	.aggregate_flowby()
	)
	return aggregated

[catherinebirney]

@matthewlchambers the bls_clean_allocation_fba_w_sec() calls on equally_allocate_suppressed_parent_to_child_naics(), which is a function used to estimate values for flow amounts that are 0 in the df - which is the way bls provides suppressed data. I can take a look at the code you provided later today or tomorrow. The function should be written generically so it can be applied to datasets other than bls

[matthewlchambers]

Once I feel sure my function is doing the right thing, then I'm happy to look at generalizing it. I just want to make sure that we're on the right track first. It was hard for me to follow equally_allocate_suppressed_parent_to_child_naics(), so I wrote the above function to do what's described in the function name. So, for each parent-children group (starting at the 2 digit level), it takes the parent total, subtracts the sum of non-suppressed children, then divides by the number of suppressed children, and attributes the result to each suppressed child. Is equally_allocate_suppressed_parent_to_child_naics() doing something beyond that?

[catherinebirney]

Your code looks right - equally_allocate_suppressed_parent_to_child_naics() isn't doing anything other than what you described.

[matthewlchambers]

Thanks, @catherinebirney! I don't think it should be too hard to generalize; I may have to come back to that task in a bit though.