Set up Rake task to clear miniprofiler logs

[orenyk]

We're in trouble:

Hi Casey and other developers:

We have a problem with the ruby webfarm backend filesystem. We are down to 4% free inodes. If that means nothing to you, suffice it to say that we have too many files on our filesystem. Not too little capacity, too many literal files.

We searched around to find where are large quantity of files came from, and reservations was the peak offender by significant margin. Within reservations, there are several folders like:

/ror_dev/apps/reservations.yale.edu/fes_vehicles/current/tmp/miniprofiler 

[root@vm-rordevweb-03 miniprofiler]# ls -1 | wc -l 
198928 

that’s a little shy of 200k files. That’s enough that when you try to list the files, first ls hangs for a minute or so before it gives you the results. There are similar quantities for the other instances of the app, so that’s more than 2 million files across reservations.

Are these files doing you any good?

It does seem like you have controls in place to not retain them forever, but the amount we are retaining is still pretty out of control, and needs to be reduced.

Is it possible to build your app without this profiler in? If you need it, can you limit to one or two instances, and not all of them?

We want to delete the majority of these files, but we’re checking with you first.

Meanwhile, there are other applications on this shared environment, and we cannot let the entire structure crash for your application. I have disabled the hudson crons on vm-rordevweb-04 until we can decide what to do.

From here it doesn't seem like it defaults to clearing out the logs, we should either set up a cron job to do it or figure out how to make it happen otherwise. I'm guessing that this is just an issue on the development instances since the profiler only runs in the development environment, but we need to fix this ASAP regardless.

[orenyk]

I'm also going to tell ITS to just clear out all of those directories; we're not using the data at the moment although we are using the profiler. Assuming that clearing the directories will buy us some time, we can then figure out how to prevent this from happening again.

[orenyk]

Quoting @dgoerger with the temporary solution for posterity:

We added env=staging to Reservations on the development server in a hacky way in order to prevent a server crash / corruption through overwriting / overloading the inodes, due to there being too many files from miniprofiler. We don't want to have to maintain hacky.

[orenyk]

I tried looking into this and got a bit stuck; ultimately we'll still want to be able to use miniprofiler in RAILS_ENV=development if possible, but right now I couldn't figure out a way to prevent MiniProfiler from generating logs to begin with. I think the easy alternative is just to add a cron job to clear out the files every day, so if I can't figure out how to tweak MiniProfiler then I'll just implement that instead. Either way, this doesn't need to be fixed in v3.4.3 since we have a hacky solution, so I'm bumping it to v4.0.

[dgoerger]

fwiw clearing the logs once per day is not enough. ITS forbade miniprofiler from its servers after it almost trashed the server by overloading inodes (my boss said it's not allowed). It's a complicated situation with the default testing suite running once every five seconds (to ensure server uptime and availability) and generating many many thousands of logs, and not wanting to disable the testing suite for a single server. Unix Infrastructure and the Dev program have a meeting to discuss a way forward next month (on the 10th I think), so, stay tuned for that, but if anyone has creative solutions please do let us know. :)

[orenyk]

Thanks for the clarification; I feel like there's a good chance we'll be restructuring our Rails environments in the end so that this is no longer an issue, but just in case I'm bumping this to v4.1.

[orenyk]

Given that we're going to revamp deployment (see #683) and are currently using the staging environment on the ITS non-production servers I don't think we'll need to specifically come up with a way to make miniprofiler play nicely in production. Closing.