I recently ran into a server running Best Practical Request Tracker (RT) version 4.0.8 that was having major performance issues displaying certain pages and causing the system to run out of resources.
What was happening was that when a user would click to view a ticket or navigate around a tickets information, it would take a few minutes for the page to load; on the server the Apache processes (httpd) would be spiked at 100% CPU usage.
Here is the output from top for the httpd process while this was occurring.
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 6593 apache 25 0 2999m 2.5g 4924 R 100.0 16.0 4:31.66 httpd
As you can see, the CPU usage is at 100% and it would sit there for a good minute or two with the occasional jump in mysql usage.
In the Apache error logs, the following entry was found (a lot of them):
[Tue Jun 10 11:01:13 2014] [error] [client 10.21.151.250] Apache2::RequestIO::rflush: (103) Software caused connection abort at /opt/perl5/lib/perl5/Plack/Handler/Apache2.pm line 153
The main effected pages (the pages that would load slow and spike the httpd process) are listed below (i’m sure there are probably more as well).
- /Ticket/Modify.html
- /Ticket/Display.html
- /Ticket/ModifyPeople.html
- /Ticket/ModifyAll.html
- /Ticket/Reminders.html
After thinking on it, I realized the common theme of these pages is they allow some level of detail or change to the owner of the ticket.
So I ran a few queries against the RT database, and found that this particular database had over 200,000 users defined, many of witch were Unicode characters or single character names, presumably created via automatic processes of some sort.
Because of these 200k users, every place in the system where the Owner (or users) are displayed (using a select list) runs very slow, and sucks up the majority of the memory on the system and pegs the httpd process.
Here are the queries you can use to see the data in the Users table in RT.
select * from Users; -- List All Users select count(*) from Users; -- Count of Users
Fixes
If you can trim down your users list, that is the best bet. If you are unable to trim your users list there are two Global RT settings that you can change that will use a look-up process (AutoComplete) to find user names instead of the select boxes. This will also “fix” the issue.
The configuration values are:
- AutocompleteOwners
- AutocompleteOwnersForSearch
The AutocompleteOwners setting can also be set per user in the user preferences.
The settings can be changed globally by adding the following to your /opt/rt4/etc/RT_SiteConfig.pm file.
Set($AutocompleteOwners, 1); Set($AutocompleteOwnersForSearch, 1);
This sets the system to, by default, use AutoComplete lists instead of select lists for the owner lists.
So if you are having issues with Apache crashing, server resource issues, or general page slow loading, and you are running RT, check to see how many users are in the system!