Sharp Tools

"Secure Connection Truncated" errors using Subversion + DAV over SSL

I was recently tasked with setting up a Subversion server using DAV+SSL, authenticating to a Windows domain controller. I fumbled about and did so, and it appeared to work okay, but when checking large trees in or out Bad Things would happen. Specifically, errors that looked like this:

svn: REPORT request failed on '/svn/repo/!svn/vcc/default' 
svn: REPORT of '/svn/repo/!svn/vcc/default': Could not read response body: Secure connection truncated (https://svn.myco.com) 

...or like this, apparently random choice as to which:

svn: REPORT request failed on '/svn/repo/!svn/vcc/default' 
svn: REPORT of '/svn/repo/!svn/vcc/default': Could not read chunk delimiter: Secure connection truncated (https://svn.myco.com) 

I had taken the following steps to try to ameliorate the problem already:

• increased the timeout to 1000 seconds (which fixed an earlier connection issue)
• tried disabling apache worker mpm and relying on prefork, as per a suggestion for someone losing connections using svn+ssh. That didn't work; re-enabled.
• disabled http compression (by commenting out the loading of mod_deflate). Left that disabled just in case.
• enabled keepalive and upped the requests per connection. No change.

Then, after investigating and watching apache while the checkout was running, I noted that the working apache process would grow during the checkout to consume appro. 27m of resident memory and then suddenly vanish when the error occured, briefly showing up in the process list as httpd <defunct>. Closer examination of the apache logs showed that these httpd children were indeed segfaulting.

This is a RHEL5 system using Subversion 1.4.5 compiled from source, being served by the distro's Apache (httpd-2.2.3-7.el5) over SSL using the distro's neon-0.25.5-5.1 at runtime AFAICT (although subversion was compiled with neon source in the tree, from subversion-deps). It is a fsfs repository on an ext3 filesystem (local disk) using mod_auth_kerb to auth to a windows DC.

THE ANSWER

So someone smarter than I continued where I left off and discovered that I was mostly right, there was a memory leak. But when I had tumbled off to get sleep, exhausted, he soldiered on, and tracked that leak to the DAV module (mod_dav_svn) in Apache. The problem, apparently, is that the DAV module was performing the authentication steps for every directory, every time it was accessed, and the leak was leaking during the auth process. Thus, in a large tree, the dozens or hundreds of auth steps would end up leaking the module into instability. The solution was to tell the DAV module not to reauthenticate for every path underneath the main one once the user had been authenticated into the repository. To do this, add the following statement to the apache config (in the vhost entry for the repo, in our case):

SVNPathAuthz off

...and in our case at least, all was well. Posted by jbz at September 14, 2007 2:29 PM | TrackBack