Gitlab 500 Error on Login

If you run your own Gitlab server sooner or later you will be confronted with the message '500 We're sorry, but something went wrong. We've been notified about this issue and we'll take a look at it shortly.'

In my set of circumstances I got this screen immediately after a 'successful' login - i.e. I logged into Gitlab with credentials I irrefutably knew were correct. Gitlab is a beast with many different components and interactions and therefore tracking down, diagnosing, and fixing the error could have been a formidable task.

The best starting point is to have a look at the /home/git/gitlab/log/production.log file.

ActionView::Template::Error (Mysql2::Error: Can't create/write to file '/var/tmp/mysql.RCx5Ky/#sql_17c5_0.MYI' (Errcode: 2): SELECT  `projects`.* FROM `projects` INNER JOIN `namespaces` ON `namespaces`.`id` = `projects`.`namespace_id` WHERE `projects`.`id` IN (1, 3, 4, 5, 6) ORDER BY projects.last_activity_at DESC LIMIT 20):
    8:           New project

This entry towards the end of the file is a good indication of the problem. MySQL cannot write to a temporary file so the problem area must be to do with MySQL. My first idea was MySQL had run out of disk space so I checked.

openSUSE-123-64-minimal:/home/git/gitlab/log # df -k
Filesystem     1K-blocks    Used Available Use% Mounted on
devtmpfs          248012      32    247980   1% /dev
tmpfs             251672       0    251672   0% /dev/shm
tmpfs             251672   26144    225528  11% /run
/dev/sda3       19091164 2415148  15706240  14% /
tmpfs             251672       0    251672   0% /sys/fs/cgroup
tmpfs             251672   26144    225528  11% /var/run
tmpfs             251672   26144    225528  11% /var/lock
/dev/sda2         516040   44848    444980  10% /boot
openSUSE-123-64-minimal:/home/git/gitlab/log #

No problems there. A little Googling and I located this thread http://serverfault.com/questions/69557/fixed-a-problem-on-a-rails-mysql-app-how-to-prevent-it-happening-again. The issue here is MySQL is closing the persistent connection opened by the Rails app. There are a number of timeout values in MySQL and as yet I am unsure which value I need to change here. I do know that I hadn't used my Gitlab server for a few weeks over the Xmas period but there had been occasions in the past when it had continued working with a few days inactivity.

The easiest workaround (for now) to get your site back up and running, is to simply reboot your MySQL server

# /etc/rc.d/mysql restart

blog terms

GitLab Linux