Bugfix CLI-select

CLI: Add remove_empty to parameter file; set to False by default (pre-v1.3.1 behaviour, and therefore backwards compatible).

Select: remove empty pairs

• Survey.select removes now empty source-receiver-frequency pairs. If you
  want the old behaviour set remove_empty=False.

• Maintenance: Added a cron to GHA; 20th of every month at 14:14.

File-based computations

• electrodes:

  • New source TxMagneticPoint (requires discretize; mainly used as
    adjoint source for magnetic receivers; does not work in the presence of
    magnetic permeabilities in the vicinity of the source).
  • Both receivers (Rx{Electric;Magnetic}Point) can now produce their
    proper adjoint (thanks to @sgkang!).

• Changes in Simulation and parallel execution.

  • Parallel computation is not sharing the simulation any longer.
  • Parallel computation can new be done both file-based or all in memory.
    The new possibility for file-based computation should make it possible
    to compute responses for any amount of source-frequency pairs. See
    parameter file_dir in the Simulation class (or corresponding parameter
    in the CLI parameter file).
  • get_model and get_hfield are now done on the fly, they are not
    stored in a dict; simulation._dict_model and
    simulation._dict_hfield do not exist any longer.
  • New methods jvec (sensitivity times a vector) and jtvec
    (sensitivity transpose times a vector). These methods are currently
    experimental; documentation and examples are lacking behind.

• Various small things:

  • Models and Fields return itself (not a copy) when the grid provided to
    interpolate_to_grid is the same as the current one.

Remove optimize & bug fix

• io: Adjustment so that hdf5 tracks the order of dicts.

• simulations:

  • Adjust printing: correct simulation results for adjusted solver printing
    levels; default solver verbosity is new 1; log can now be
    overwritten in solver_opts (mainly for debugging).

  • Functions moved out of simulations: expand_grid_model moved to
    models and estimate_gridding_options to meshes. The
    availability of these functions through simulations will be removed in
    v1.4.0.

• optimize: the module is deprecated and will be removed in v1.4.0. The two
  functions optimize.{misfit;gradient} are embedded directly in
  Simulation.{misfit;gradient}.

White noise

• CLI:

  • New parameters save and load to save and load an entire simulation.
    In the parameter file, they are under [files]; on the command line,
    they are available as --save and --load; they are followed by the
    filename including its path and suffix. (In turn, the parameter
    store_simulation was removed.)

• simulations.Simulation:

  • Warns if the gradient is called, but receiver_interpolation is not
    'linear'.
  • Slightly changed the added noise in compute(observed=True): It uses new
    the survey.add_noise attribute. There is new a flag to set if noise
    should be added or not (add_noise), and if the amplitudes should be
    chopped or not (min_amplitude). Also note that the added noise is new
    white noise with constant amplitude and random phase.

• surveys:

  • New function random_noise, which can be used to create random noise in
    different ways. The default noise is white noise, hence constant amplitude
    with random phase. (This is different to before, where random Gaussian
    noise was added separately to the real and imaginary part.) For the random
    noise it requires new at least NumPy 1.17.0.

  • New attribute Survey.add_noise, which uses under the hood above
    function.

  • A Survey can new be instantiated without receivers by setting
    receivers to None. This is useful if one is only interested in
    forward modelling the entire fields. In this case, the related data object
    and the noise floor and relative error have no meaning. Also, in
    conjunction with a Simulation, the misfit and the gradient will be zero.

• Various:

  • All emg3d-warnings (not solver warnings) are now set to 'always', and
    corresponding print statements were removed.
  • Simplified (unified) _edge_curl_factor (private fct).

Adjoint-fix for electric receivers

This release contains, besides the usual small bugfixes, typos, and small
improvements, an important fix for optimize.gradient. Keep in mind that
while the forward modelling is regarded as stable, the optimize module is
still work in progress.

The fixes with regard to optimize.gradient ensure that the gradient is
indeed using the proper adjoint to back-propagate the field. This is currently
only given for electric receivers, not yet for magnetic receivers. These
improvement happened mainly thanks to the help of Seogi (@sgkang).

The changes in more detail:

• fields:

  • get_receiver has a new keyword method, which can be 'cubic' or
    'linear'; default is the former, which is the same behaviour as before.
    However, if you want to compute the gradient, you should set it to
    'linear' in your Simulation parameters. Otherwise the adjoint-state
    gradient will not exactly be the adjoint state.
  • get_source_field returns new the real-valued, frequency-independent
    source vector if frequency=None.
  • get_source_field uses the adjoint of trilinear interpolation for point
    sources (new). For dipoles and wires it the source is distributed onto the
    cells as fraction of the source length (as before).

• electrodes: Re-introduced the point source as TxElectricPoint.

• simulations.Simulation:

  • New keyword receiver_interpolation, which corresponds to the method
    in get_receiver (see above). Cubic is more precise. However, if you are
    interested in the gradient, you need to choose linear interpolation at the
    moment, as the point source is the adjoint of linear interpolation. To be
    the proper adjoint for the gradient the receiver has to be interpolated
    linearly too.
  • If gridding is 'same' or 'input', it checks now if the provided
    grid is a sensible grid for emg3d; if not, it throws a warning.

• meshes: New function check_grid to verify if a given grid is good for
  emg3d.

• optimize.gradient: Changed order when going from computational grid to
  inversion grid. Changing the grids at the field stage (cubic interpolation)
  seems to be better than changing at the cell-averaged stage::

  New: field_comp -> field_inv -> cells_inv
  Old: field_comp -> cells_comp -> cells_inv
  

• cli: Uses now by default linear receiver interpolation if the
  gradient is wanted (new), otherwise it uses cubic interpolation (as
  before). The new keyword receiver_interpolation of the simulation can be
  set in the parameter file, which overwrites the described default behaviour.

Stable API

Here it is, three months of hard labour lead to v1.0.0!

There are many changes, and they are listed below for each module.

Your existing code will break, and I apologize for it. Please do not hesitate
to get in touch if you have troubles updating your code.

API: With version 1.0 the API becomes stable and you can expect that your
code will work fine for the duration of emg3d v1.x.

• Removed all deprecated features.
• Reduced top namespace to principal functions; get_receiver is not in the
  top namespace any longer. It is advised to use directly the field method:
  field.get_receiver.
• Moved emsig.github.io to emsig.xyz and emsig.readthedocs.io to
  emg3d.emsig.xyz.
• Changed principal repo branch from master to main.

Detailed changes by module

CLI

• Because frequencies are now dicts as well in a Survey they have to be named
  by their key instead of their value when selecting data in the parameter
  file.
• Entire configuration is now added to the log file.

Core

• restrict_weights: New signature.

Electrodes

• New module containing all sources and receivers. Currently implemented are
  TxElectricDipole, TxMagneticDipole, TxElectricWire,
  RxElectricPoint, and RxMagneticPoint.
• New class TxElectricWire for an arbitrary electric wire.
• Receivers can be defined in absolute coordinates, or in coordinates relative
  to source position if they move with the source. Latter makes only sense
  within a Survey/Simulation.
• dip is new called elevation to make it clear that it is the angle
  positive upwards (anticlockwise from the horizontal plane).
• Bugfix of the loop area for a magnetic dipole (the area was previously wrong
  except for dipoles of length of 1).
• Zero source strength does no longer mean "normalized", it means zero
  strength (hence no source).
• Besides the sources and receivers it contains utilities how to move
  electrodes in the coordinate system (e.g., rotation).

Fields

• fields.Field:

  • Is not a subclassed ndarray any longer; with all its advantages and
    disadvantages. E.g., operations on Field are not possible any longer
    and have to be carried out on Field.field. However, it should be easier
    to maintain and expand in the future.
  • New signature.
  • Knows new its grid. As a consequence, all functions that required
    previously the grid and the field require new only the field;
    e.g., emg3d.fields.get_receiver.
  • Has no property ensure_pec any longer, it is ensured directly in
    solver.prolongation.
  • Has new the methods interpolate_to_grid and get_receiver.

• Renamed parameters in all functions:

  • src to source;
  • freq to frequency;
  • rec to receiver.

• Removed functions and classes:

  • SourceField; it is just a regular Field now;
  • get_receiver (the name still exists, but it is now what was before
    fields.get_receiver_response).

• Renamed functions and classes (both do not take a grid any longer):

  • get_h_field to get_magnetic_field;
  • fields.get_receiver_response to fields.get_receiver.

I/O

• Model, Field, Survey, and Simulation instances saved with an
  older version of emg3d will not be able to de-serialize with version 1.0. You
  have to update those files, see this gist:
  https://gist.github.com/prisae/8345c3798e35f1c73efef617ac495538

Maps

• Changed function and class names:

  • _Map to BaseMap;
  • grid2grid to interpolate (new signature);
  • edges2cellaverages to interp_edges_to_vol_averages (new signature);
  • volume_average to interp_volume_average (new signature);
  • interp3d to interp_spline_3d (new signature).

• maps.interpolate:

  • Can now be used to interpolate values living on a grid to another grid or
    to points defined either by a tuple or by an ndarray.
  • The implemented interpolation methods are 'nearest' (new), 'linear',
    'cubic', and 'volume'. Volume averaging ('volume') only works for
    grid-to-grid interpolations, not for grid-to-points interpolations.
  • Does not accept entire fields any longer. Entire fields can be mapped with
    their own field.interpolate_to_grid method.

• Maps cannot be (de-)serialized any longer ({to;from_dict}); simply store
  its name, which can be provided to models.Model.

• Function rotation should be used for anything involving angles to use
  the defined coordinate system consistently.

Meshes

• Changed function and class names:

  • _TensorMesh to BaseMesh;
  • min_cell_width to cell_width.
  • get_origin_widths to origin_and_widths (has new finer loops to fine
    grid sizes than before).

• meshes.BaseMesh:

  • Reduced to the attributes origin, h, shape_{cells;nodes},
    n_{cells;edges;faces}, n_{edges;faces}_{x;y;z},
    {nodes;cell_centers}_{x;y;z}, shape_{edges;faces}_{x;y;z}, and
    cell_volumes. These are the only required attributes for emg3d.

• meshes.construct_mesh: domain, vector, distance,
  stretching, min_width_limits, and min_width_pps can now also
  be provided as a dict containing the three keys '{x;y;z}'.

• meshes.skin_depth takes new mu_r instead of mu.

• good_mg_cell_nr: max_prime is new max_lowest, as it could also
  be, e.g., 9, which is not a prime.

Models

• models.Model:

  • Knows new its grid. As a consequence, all the functions that used to
    require the grid and the model require new only the model;
    e.g., emg3d.solver.solve or emg3d.fields.get_magnetic_field.

  • If property_y or property_z are not set they return now None,
    not property_x.

  • If a float is provided for a property it is new expanded to the shape of
    the model, and not kept as a float.

  • Has to be initiated with all desired properties; it cannot be changed
    afterwards. E.g., if it was initiated without electric permittivity, it
    cannot be added afterwards. However, it can be initiated with dummy values
    and adjusted later.

  • Renamed interpolate2grid to interpolate_to_grid.

• models.VolumeModel: Does not take a grid any longer.

Simulations

• Simulation:

  • Works new for electric and magnetic dipole sources as well as electric wire
    sources; electric and magnetic point receivers.
  • Works now as well for surveys that contain receivers which are positioned
    relatively to the source.
  • New signature: no grid any longer, name is new an optional keyword
    parameter, new optional keyword parameter info.
  • Method get_sfield is removed.

• expand_grid_model and estimate_gridding_opts have new signatures and
  do not take a grid any longer.

Solver

• solver.solve:

  • New signature: no grid any longer; efield and cycle are moved
    to keyword arguments.

  • The defaults for sslsolver, semicoarsening, and linerelaxation
    is new True (before it was False). This is not necessarily the
    fastest setting, but generally the most robust setting.

  • New keyword parameter plain, which is by default False. If it is
    set to True it uses plain multigrid, hence sslsolver=False,
    semicoarsening=False, and linerelaxation=False, unless these
    parameters were set to anything different than True.

  • Some verbosity levels changed (for consistency reasons throughout emg3d).
    The new levels are [old levels in brackets]:

    • -1: Nothing [0]
    • 0: Warnings [1]
    • 1: One-liner at the end [2]
    • 2: One-liner (dynamically updated) [-1]
    • 3: Runtime and information about the method [same]
    • 4: Additional information for each MG-cycle [same]
    • 5: Everything (slower due to additional error computations) [same]

    Level three updates now dynamically, just as level 2.

• solve_source(): New function, a shortcut for solve(). It takes a
  source and a frequency instead of a sfield, gets the sfield
  internally, and forwards everything to solver.solve.

• multigrid, krylov, smoothing, restriction, prolongation,
  residual, RegularGridProlongator: New signature, mainly not taking a
  grid any longer.

Surveys

• Survey:

  • frequencies is new a dict just like sources and receivers.
  • sources and receivers must be tuples or dicts; lists are no longer
    permitted. For this, the module surveys has new convenience functions
    txrx_coordinates_to_dict and txrx_lists_to_dict.
  • Has no attribute observed any longer; access it just like any other
    data through Survey.data.observed.
  • rec_coords and src_coords attributes changed to the methods
    receiver_coordinates and source_coordinates.
    receiver_coordinates takes an optional source key.
    For relatively located receivers, it returns by default all positions of
    this receiver for all source position. If a source-key is provided it only
    returns the receiver position for this source. This does not affect
    absolutely positioned receivers.
  • Has no attribute rec_types any longer.
  • name is new optional.
  • New optional keywords date and info.
  • noise_floor and relative_error are new stored as data array if they
    are not floats.
  • The keyword fixed has been dropped. To simulate fixed surveys define
    the receivers with a relative offs...

Magnetics in Simulation

• Simulation:

  • Sources and receivers can now be magnetic, also for the adjoint-state
    gradient (unit loops, not yet arbitrarily loops).

• fields.get_source_field:

  • The msrc argument introduced in v0.16.0 is renamed to electric, and
    has the opposite meaning. If True, the source is electric, if False, the
    source is magnetic. This was made to streamline the meaning with the
    meaning given in surveys.Dipole. The old parameter msrc is
    deprecated and will be removed. Warning, if msrc was provided as
    positional argument instead of as keyword argument it will now be taken as
    electric, with the opposite meaning (backwards incompatible).
  • The magnetic source was corrected and has the opposite sign now (factor -1;
    backwards incompatible).

• Bug fixes:

  • Simulation: Stop overwriting synthetic data if provided in the survey to a
    simulation.
  • CLI: Removed configuration info from output data; caused problems when
    storing to h5. This has to be resolved with properly addressing the io
    stuff. Currently only stores the data selection to output data.

Verbosity & Logging

• Solve has a new keyword log, which enables to log the solver messages
  in the returned info dictionary instead of printing them to screen. This is
  utilized in the CLI and in the Simulation class to log the solver info.

• Survey has a new attribute select, which returns a reduced survey
  containing the selected sources, receivers, and frequencies.

• CLI:

  • Configuration info is added to output data.
  • Checks now first if all required files and directories exist, and exits
    gracefully otherwise informing the user. (The default thrown Python errors
    would be good enough; but user of the CLI interface might not be familiar
    with Python, so it is better to throw a very simple, clear message.)
  • Log is more verbose with regards to solver (rel. error, time, nr of it.).

• Dipole throws new an error instead of a warning if it received an unknown
  keyword.

• Various small things with regard to how things are logged or shown on screen.

• Changed all DeprecationWarnings to FutureWarnings, meaning they will
  be removed in the next release.

• Bug fix with regards to data selection in the CLI; moved to Survey (see
  above).

Arbitrarily shaped sources

• fields.get_source_field:

  • Arbitrarily shaped sources (and therefore also loops) can now be created by
    providing a src that consists of x-, y-, and z-coordinates of all
    endpoints of the individual segments.

  • Simple "magnetic dipole" sources can now be created by providing a point
    dipole ([x, y, z, azm, dip]) and set msrc=True. This will create a
    square loop of lengthxlength m perpendicular to the defined point
    dipole, hence simulating a magnetic source. Default length is 1 meter.

  • Point dipoles and finite length dipoles were before treated differently.
    Point dipoles are new converted into finite length dipoles of provided
    length (default is 1 meter), and treated as finite length dipoles. This is
    backwards incompatible and means that the source field for point dipoles
    might not be exactly the same as before. However, in any properly set-up
    simulation this should have no influence on the result.

  • Bugfix: Fix floating point issue when the smaller coordinate of a finite
    length dipole source was very close to a node, but not exactly. This is
    done by rounding the grid locations and source position, and the precision
    can be controlled via decimals; default is micrometer.

• fields: Values outside the grid in get_receiver and
  get_receiver_response are new set to NaN's instead of zeroes.
  Additionally, the first and last values in each direction of the fields are
  ignored, to avoid effects form the boundary condition (receivers should not
  be placed that close to the boundary anyway).

• simulations:

  • Within the automatic gridding the properties are estimated much more
    conservative now, if not provided: before the log10-average of the last
    slice in a given direction was used; now it uses the maximum resistivity.
    This is usually the air value for x/y and positive z. This is very
    conservative, but avoids that users use too small computational domains in
    the case of land and shallow marine surveys. The downside is that it
    heavily over-estimates the required domain in the deep marine case.
    However, slower but safe is better in this case.
  • New method print_grids, which prints the info of all created grids.
    This is also used for logging in the CLI interface.

• maps: interp3d takes a new keyword cval, which is passed to
  map_coordinates.