Exporting variables from root, hidden and secure

[jamesrkg]

This ticket is partially influenced the the conversation here:

https://yougov.slack.com/messages/C2MLN3SNS/convo/C2MLN3SNS-1561587638.058400/

Currently, ds.export's hidden and variables kwargs are mutually exclusive and trigger this code:

scrunch/scrunch/datasets.py

Lines 2217 to 2231 in c857398

	# hidden is mutually exclusive with
	# variables to include in the download
	if hidden and not variables:
	if not self.resource.body.permissions.edit:
	raise AttributeError(
	"Only Dataset editors can export hidden variables")
	payload['where'] = {
	'function': 'select',
	'args': [{
	'map': {
	x: {'variable': x}
	for x in self.resource.variables.index.keys()
	}
	}]
	}

With the addition of the secure tree this is now outdated, since hidden=True currently also exports all secure variables as well though based on the above linked conversation that is likely to change on the server side. We should update this in so doing improve flexibility and control overall.

Current behavior:

• variables
  • None - as controlled by the API (this will export non-hidden and non-secure variables)
  • list - aliases of variables to export, these can be any variables be they hidden, secure or not. Passing a list here will also supersede hidden, which will be ignored.
• hidden
  • True - export all hidden variables, will be ignored if a list is passed to variables.
  • False - do not export the hidden variables, will be ignored if a list is passed to variables.

Goals:

• Remove mutual exclusivity between variables and hidden kwargs
• Add secure kwarg (or private??)
• Support simple way to export some, none or all non-hidden, non-secure variables
• Support simple way to export some, none or all hidden variables
• Support simple way to export some, none or all secure variables
• With a combination of the above, throw clear error if no variables have been requested
• Implementing the above will be simplified by the recent addition of hidden and secure attributes to the variables catalog.
• Safeguard the export of hidden and/or secure variables in principle, separate to whether or not they are explicitly requested (since the location of these variables can change over time).

Proposal:

• variables
  • None - as controlled by the API (this will export non-hidden and non-secure variables)
  • list - aliases of variables to export, these can be any variables be they hidden, secure or neither.
• hidden
  • None - no hidden variables will be exported, even if they are in a list passed to variables.
  • True - automatically export all hidden variables, in addition to any other variables passed as a list passed to variables
  • False - do not automatically export all hidden variables but hidden variables in a list passed to variables will be exported.
• secure
  • None - no secure variables will be exported, even if they are in a list passed to variables.
  • True - automatically export all secure variables, in addition to any other variables passed as a list passed to variables
  • False - do not automatically export all secure variables but secure variables in a list passed to variables will be exported.