Deploying with Amnesia on prod

[ak-iz]

I have implemented Amnesia on my existing project. It works fine on local env but not on prod. I am deploying it using mix release through docker/kubernetes.

This is what I have put inside config.exs:

config :mnesia,
  dir: '.mnesia/#{Mix.env()}/#{node()}'

I run mix amnesia.create -d Database --disk! through docker script but then I am not sure where to put or how the database disks are formed. And it is also evident from the error I am getting when I run the server, it crashes:

09:40:39.212 [error] Task #PID<0.4952.0> started from ClientApi.Supervisor terminating
** (stop) {:aborted, {:no_exists, Database.InValidRssCache}}
    (mnesia 4.18) mnesia.erl:361: :mnesia.abort/1
    (mnesia 4.18) mnesia.erl:1794: :mnesia.do_dirty_write/3
    (client_api 0.0.1) lib/client_api/amensia/database.ex:32: Database.InValidRssCache.write!/1
    (client_api 0.0.1) lib/client_api/invalid_rss_cache.ex:14: ClientApi.InvalidRssCache.write!/2
    (elixir 1.11.2) lib/enum.ex:786: Enum."-each/2-lists^foreach/1-0-"/2
    (client_api 0.0.1) lib/client_api/itunes/cache_builder.ex:15: ClientApi.Itunes.CacheBuilder.build/0
    (elixir 1.11.2) lib/task/supervised.ex:90: Task.Supervised.invoke_mfa/2
    (stdlib 3.13.2) proc_lib.erl:226: :proc_lib.init_p_do_apply/3
Function: &ClientApi.CacheInitializer.run/0
    Args: []

09:40:39.220 [info] Application client_api exited: shutdown
{"Kernel pid terminated",application_controller,"{application_terminated,client_api,shutdown}"}
Kernel pid terminated (application_controller) ({application_terminated,client_api,shutdown})

Any help on this?

[noizu]

Are any tables available? if not you can console_clean and schema destroy schema create, make sure mount points exist. Amnesia.info(:all)

[ak-iz]

Are any tables available? if not you can console_clean and schema destroy schema create, make sure mount points exist. Amnesia.info(:all)

@noizu Yes, tables do exist. I have CacheInitializer module that starts writing to tables as soon as the application starts, so my app crashes as soon as it starts.

[noizu]

You can get into a weird state with schema sometimes where bags and certain disk based ones will throw up on you. I usually just do console clean and recover that way.

for extreme scenarios I sometime use this but most likely if you don't have existing critical data destroying the schema or the table and recreating after checking config should help.

I haven't run this in quite a while not sure if the current version is complete so oyu'd need to get into the mnesia undocumented weeds to run this.

:ets.insert(:mnesia_gvar, {{MyAccount.Database.TableToRemove, :where_to_read}, [node()]} )
:ets.insert(:mnesia_gvar, {{MyAccount.Database.TableToRemove, :where_to_write}, [node()]} )
tables = [
  MyAccount.Database.TableToRemove
]
r = for tab <- tables do
  IO.puts "Force removing #{tab}"
  :mnesia_schema.schema_transaction(fn() ->
    IO.puts "Transaction . . ."
    tid_ts = :mnesia_schema.get_tid_ts_and_lock(:schema, :write)
    IO.puts "Transaction . . . lock: #{inspect tid_ts}"
    case :ets.lookup_element(:mnesia_gvar, {:schema, :where_to_write}, 2) do
      [] -> :mnesia.abort({:read_only, tab});
      _ -> :ok
    end
    IO.puts "Lock table"
    #tab_lock = :mnesia_schema.get_tid_ts_and_lock(tab, :write)
    #IO.puts "Transaction . . . table lock: #{inspect tab_lock}"
    cs = :ets.lookup_element(:mnesia_gvar, {tab, :cstruct}, 2)
    IO.puts "Got Object: #{inspect cs} . . insure active"
    #:mnesia_schema.ensure_active(cs)
    IO.puts "Looking up internals . . ."
    lookup = case :ets.lookup_element(:mnesia_gvar, {tab, :where_to_write}, 2) do
      [] ->
           IO.puts "No where to write"
           :mnesia.abort({:read_only, tab});
      v -> v
    end
    IO.puts "Internals: #{inspect lookup}"
    IO.puts "MODIFY: #{inspect cs, pretty: true, limit: :infinity}"
    op = {:op, :delete_table, :mnesia_schema.cs2list(cs)}
    :mnesia_schema.insert_schema_ops(tid_ts, [op])
    #:mnesia.abort({:read_only, tab});
  end)
end

[ak-iz]

You can get into a weird state with schema sometimes where bags and certain disk based ones will throw up on you. I usually just do console clean and recover that way.

for extreme scenarios I sometime use this but most likely if you don't have existing critical data destroying the schema or the table and recreating after checking config should help.

I haven't run this in quite a while not sure if the current version is complete so oyu'd need to get into the mnesia undocumented weeds to run this.

:ets.insert(:mnesia_gvar, {{MyAccount.Database.TableToRemove, :where_to_read}, [node()]} )
:ets.insert(:mnesia_gvar, {{MyAccount.Database.TableToRemove, :where_to_write}, [node()]} )
tables = [
  MyAccount.Database.TableToRemove
]
r = for tab <- tables do
  IO.puts "Force removing #{tab}"
  :mnesia_schema.schema_transaction(fn() ->
    IO.puts "Transaction . . ."
    tid_ts = :mnesia_schema.get_tid_ts_and_lock(:schema, :write)
    IO.puts "Transaction . . . lock: #{inspect tid_ts}"
    case :ets.lookup_element(:mnesia_gvar, {:schema, :where_to_write}, 2) do
      [] -> :mnesia.abort({:read_only, tab});
      _ -> :ok
    end
    IO.puts "Lock table"
    #tab_lock = :mnesia_schema.get_tid_ts_and_lock(tab, :write)
    #IO.puts "Transaction . . . table lock: #{inspect tab_lock}"
    cs = :ets.lookup_element(:mnesia_gvar, {tab, :cstruct}, 2)
    IO.puts "Got Object: #{inspect cs} . . insure active"
    #:mnesia_schema.ensure_active(cs)
    IO.puts "Looking up internals . . ."
    lookup = case :ets.lookup_element(:mnesia_gvar, {tab, :where_to_write}, 2) do
      [] ->
           IO.puts "No where to write"
           :mnesia.abort({:read_only, tab});
      v -> v
    end
    IO.puts "Internals: #{inspect lookup}"
    IO.puts "MODIFY: #{inspect cs, pretty: true, limit: :infinity}"
    op = {:op, :delete_table, :mnesia_schema.cs2list(cs)}
    :mnesia_schema.insert_schema_ops(tid_ts, [op])
    #:mnesia.abort({:read_only, tab});
  end)
end

@noizu I ran your code but it had the same behavior. Then I created a release myself and consoled :mnesia.info before it crashes. I got this. It seems tables exist and they are being used.

---> Processes holding locks <--- 
---> Processes waiting for locks <--- 
---> Participant transactions <--- 
---> Coordinator transactions <---
---> Uncertain transactions <--- 
---> Active tables <--- 
schema         : with 8        records occupying 1265     words of mem
===> System info in version "4.18.1", debug level = none <===
opt_disc. Directory "/Users/vbu-01/Projects/someeee/.mnesia/prod/nonode@nohost" is used.
use fallback at restart = false
running db nodes   = [client_api@localhost]
stopped db nodes   = [nonode@nohost] 
master node tables = []
remote             = ['Elixir.Database','Elixir.Database.DayCache',
                      'Elixir.Database.DirectoryCache',
                      'Elixir.Database.EpisodeCache',
                      'Elixir.Database.EventCache',
                      'Elixir.Database.InValidRssCache',
                      'Elixir.Database.RssCache']
ram_copies         = [schema]
disc_copies        = []
disc_only_copies   = []
[] = ['Elixir.Database','Elixir.Database.EventCache',
      'Elixir.Database.RssCache','Elixir.Database.InValidRssCache',
      'Elixir.Database.DirectoryCache','Elixir.Database.DayCache',
      'Elixir.Database.EpisodeCache']
[{client_api@localhost,ram_copies}] = [schema]
2 transactions committed, 0 aborted, 0 restarted, 0 logged to disc
0 held locks, 0 in queue; 0 local transactions, 0 remote
0 transactions waits for other nodes: []

amnesia: :ok
{"Kernel pid terminated",application_controller,"{application_terminated,client_api,shutdown}"}
Kernel pid terminated (application_controller) ({application_terminated,client_api,shutdown})

Crash dump is being written to: erl_crash.dump...done

I commented the CacheInitializer.run() and still it crashed.

[noizu]

If you're using distillery start with console_clean if you're using mix release you can edit the release bin/proj-name file like below to start an instance with out loading applications and then service commands (or run them directly with eval if using mix release.)

I use distillery so not sure if that would be an identical experience.

The tables are owned by a different node it looks like. you can do TableName.info(:all) to view the internal status and where it thinks it belongs. if they are created against a different node name that would cause issues.

[noizu]

running db nodes = [client_api@localhost]
stopped db nodes = [nonode@nohost]

[zohaib-shamshad]

@noizu I faced a problem similar to what user @ak-iz talked about. I followed the advice given by you, and it helped me understand the issue but after start_clean I did not see my project running or any of the amnesia nodes starting.

On a side note, I have another question because I'm a bit confused about something. Here's what's going on:

When I use the command 'mix release', it makes a file called 'bin/<project_name>'. This file has a function called 'start' but doesn't have 'start_clean'. Later, I run a script in Docker. I want to know how I can set things up so that when I run the Docker script, I don't need to manually add 'start_clean' to that file. Can you help me with this as well?

[ak-iz]

@zohaib-shamshad @noizu Yeah, same issue here. Running Amnesia.start() and running the app after start_clean gives the same crash error.