Gitlab 500 Error on Login
If you run your own Gitlab server sooner or later you will be confronted with the message '500 We're sorry, but something went wrong. We've been notified about this issue and we'll take a look at it shortly.'
In my set of circumstances I got this screen immediately after a 'successful' login - i.e. I logged into Gitlab with credentials I irrefutably knew were correct. Gitlab is a beast with many different components and interactions and therefore tracking down, diagnosing, and fixing the error could have been a formidable task.
The best starting point is to have a look at the /home/git/gitlab/log/production.log file.
ActionView::Template::Error (Mysql2::Error: Can't create/write to file '/var/tmp/mysql.RCx5Ky/#sql_17c5_0.MYI' (Errcode: 2): SELECT `projects`.* FROM `projects` INNER JOIN `namespaces` ON `namespaces`.`id` = `projects`.`namespace_id` WHERE `projects`.`id` IN (1, 3, 4, 5, 6) ORDER BY projects.last_activity_at DESC LIMIT 20):
8: New project
This entry towards the end of the file is a good indication of the problem. MySQL cannot write to a temporary file so the problem area must be to do with MySQL. My first idea was MySQL had run out of disk space so I checked.
openSUSE-123-64-minimal:/home/git/gitlab/log # df -k
Filesystem 1K-blocks Used Available Use% Mounted on
devtmpfs 248012 32 247980 1% /dev
tmpfs 251672 0 251672 0% /dev/shm
tmpfs 251672 26144 225528 11% /run
/dev/sda3 19091164 2415148 15706240 14% /
tmpfs 251672 0 251672 0% /sys/fs/cgroup
tmpfs 251672 26144 225528 11% /var/run
tmpfs 251672 26144 225528 11% /var/lock
/dev/sda2 516040 44848 444980 10% /boot
No problems there. A little Googling and I located this thread http://serverfault.com/questions/69557/fixed-a-problem-on-a-rails-mysql-app-how-to-prevent-it-happening-again. The issue here is MySQL is closing the persistent connection opened by the Rails app. There are a number of timeout values in MySQL and as yet I am unsure which value I need to change here. I do know that I hadn't used my Gitlab server for a few weeks over the Xmas period but there had been occasions in the past when it had continued working with a few days inactivity.
The easiest workaround (for now) to get your site back up and running, is to simply reboot your MySQL server
# /etc/rc.d/mysql restart