memory growing when using SSL connections

[David]

Hi,

I'm using tn3270 sessions running over SSL. I may have up to 124 sessions activated concurrently, although I plan to get up to 250 sessions at some point.

Whenever the sessions are stopped and restarted, I notice intermittently that memory grows in multiples of 4K bytes.

I'm running on AIX 5.1, 5.2 and 5.3 and using openssl-0.9.8l.
There doesn't appear to be an obvious memory leak in either my application or the OpenSSL stuff (all memory allocated when the sessions are started are freed when the sessions are stopped).
Here's a summary of the code structure:

SSL_library_init();
meth = TLSv1_client_method();
RAND_seed();
ctx = SSL_CTX_new(meth);

while ([some telnet connection wants to do SSL])
{
ssl = SSL_new(ctx);
SSL_set_fd()
SSL_set_cipher_list();
SSL_set_connect_state();
SSL_connect();
do SSL_read(), SSL_write()
SSL_shutdown();
close FD;
SSL_free();
CRYPTO_cleanup_all_ex_data();
}

Any ideas would be appreciated.
Thanks,
David

[Jeremy Hunt]

Hi David,

Off the top of my head I notice that you do not call this as part of the
cleanup: SSL_CTX_free() [
http://www.openssl.org/docs/ssl/SSL_CTX_free.html# ]

There is also this: [ http://www.openssl.org/support/faq.html#PROG13 ]

Regards,

Jeremy

*[safeTgram (safetgram-in) receive status: NOT encrypted, NOT signed.]*
Hi,
I'm using tn3270 sessions running over SSL. I may have up to 124
sessions activated concurrently, although I plan to get up to 250
sessions at some point.
Whenever the sessions are stopped and restarted, I notice
intermittently that memory grows in multiples of 4K bytes.
I'm running on AIX 5.1, 5.2 and 5.3 and using openssl-0.9.8l.
There doesn't appear to be an obvious memory leak in either my
application or the OpenSSL stuff (all memory allocated when the
sessions are started are freed when the sessions are stopped).
SSL_library_init();
meth = TLSv1_client_method();
RAND_seed();
ctx = SSL_CTX_new(meth);
while ([some telnet connection wants to do SSL])
{
ssl = SSL_new(ctx);
SSL_set_fd()
SSL_set_cipher_list();
SSL_set_connect_state();
SSL_connect();
do SSL_read(), SSL_write()
SSL_shutdown();
close FD;
SSL_free();
CRYPTO_cleanup_all_ex_data();
}
Any ideas would be appreciated.
Thanks,
David

[Jeremy Hunt]

Hi David,

After reading this
[http://www.nlnetlabs.nl/downloads/publications/hsm/hsm_node21.html] ,
it occurs to me that you might need EVP_cleanup() too. You might
consider his clean_up() routine in general, or at least consider his
comments on it.

I think it is true to say that for every function in openSSL that sets
up structures etc there is a corresponding free function.

Regards,

Jeremy

*[safeTgram (safetgram-in) receive status: NOT encrypted, NOT signed.]*
Hi,
I'm using tn3270 sessions running over SSL. I may have up to 124
sessions activated concurrently, although I plan to get up to 250
sessions at some point.
Whenever the sessions are stopped and restarted, I notice
intermittently that memory grows in multiples of 4K bytes.
I'm running on AIX 5.1, 5.2 and 5.3 and using openssl-0.9.8l.
There doesn't appear to be an obvious memory leak in either my
application or the OpenSSL stuff (all memory allocated when the
sessions are started are freed when the sessions are stopped).
SSL_library_init();
meth = TLSv1_client_method();
RAND_seed();
ctx = SSL_CTX_new(meth);
while ([some telnet connection wants to do SSL])
{
ssl = SSL_new(ctx);
SSL_set_fd()
SSL_set_cipher_list();
SSL_set_connect_state();
SSL_connect();
do SSL_read(), SSL_write()
SSL_shutdown();
close FD;
SSL_free();
CRYPTO_cleanup_all_ex_data();
}
Any ideas would be appreciated.
Thanks,
David

______________________________________________________________________
OpenSSL Project http://www.openssl.org
User Support Mailing List openssl-***@openssl.org
Automated List Manager ***@openssl.org

[David]

Jeremy,

Thanks for the comments. The ctx stays around for the life of the
process, so I never free it unless the process is about to exit. Only the
ssl connections are freed and new ones are created. There are multiple ssl
connections per ctx. I've actually tried using one ssl connection per ctx
and freeing both ssl and ctx after use, but the results didn't change.

Regards,
David

--------------------------------------------------
From: "Jeremy Hunt" <***@optimation.com.au>
Sent: Thursday, January 07, 2010 5:20 PM
To: <openssl-***@openssl.org>
Cc: <***@bellsouth.net>
Subject: Re: memory growing when using SSL connections

Post by Jeremy Hunt
Hi David,
Off the top of my head I notice that you do not call this as part of the
cleanup: SSL_CTX_free() [
http://www.openssl.org/docs/ssl/SSL_CTX_free.html# ]
There is also this: [ http://www.openssl.org/support/faq.html#PROG13 ]
Regards,
Jeremy

*[safeTgram (safetgram-in) receive status: NOT encrypted, NOT signed.]*
Hi,
I'm using tn3270 sessions running over SSL. I may have up to 124
sessions activated concurrently, although I plan to get up to 250
sessions at some point. Whenever the sessions are stopped and restarted,
I notice intermittently that memory grows in multiples of 4K bytes.
I'm running on AIX 5.1, 5.2 and 5.3 and using openssl-0.9.8l. There
doesn't appear to be an obvious memory leak in either my application or
the OpenSSL stuff (all memory allocated when the sessions are started are
freed when the sessions are stopped).
SSL_library_init();
meth = TLSv1_client_method();
RAND_seed();
ctx = SSL_CTX_new(meth);
while ([some telnet connection wants to do SSL])
{
ssl = SSL_new(ctx);
SSL_set_fd()
SSL_set_cipher_list(); SSL_set_connect_state();
SSL_connect();
do SSL_read(), SSL_write()
SSL_shutdown();
close FD;
SSL_free();
CRYPTO_cleanup_all_ex_data();
}
Any ideas would be appreciated.
Thanks,
David

--
"The most likely way for the world to be destroyed, most experts agree, is
by accident. That's where we come in; we're computer professionals. We
cause accidents." -- Nathaniel Borenstein, co-creator of MIME
______________________________________________________________________
OpenSSL Project http://www.openssl.org

______________________________________________________________________
OpenSSL Project http://www.openssl.org
User Support Mailing List openssl-***@openssl.org
Automated List Manager ***@openssl.org

[David]

Jeremy,
I'm not sure if I've tried the EVP_cleanup() routine. I'll give it a shot.

Thanks,
David

--------------------------------------------------
From: "Jeremy Hunt" <***@optimation.com.au>
Sent: Thursday, January 07, 2010 5:34 PM
To: <openssl-***@openssl.org>; <***@bellsouth.net>
Subject: Re: memory growing when using SSL connections

Post by Jeremy Hunt
Hi David,
After reading this
[http://www.nlnetlabs.nl/downloads/publications/hsm/hsm_node21.html] , it
occurs to me that you might need EVP_cleanup() too. You might consider
his clean_up() routine in general, or at least consider his comments on
it.
I think it is true to say that for every function in openSSL that sets up
structures etc there is a corresponding free function.
Regards,
Jeremy

*[safeTgram (safetgram-in) receive status: NOT encrypted, NOT signed.]*
Hi,
I'm using tn3270 sessions running over SSL. I may have up to 124
sessions activated concurrently, although I plan to get up to 250
sessions at some point. Whenever the sessions are stopped and restarted,
I notice intermittently that memory grows in multiples of 4K bytes.
I'm running on AIX 5.1, 5.2 and 5.3 and using openssl-0.9.8l. There
doesn't appear to be an obvious memory leak in either my application or
the OpenSSL stuff (all memory allocated when the sessions are started are
freed when the sessions are stopped).
SSL_library_init();
meth = TLSv1_client_method();
RAND_seed();
ctx = SSL_CTX_new(meth);
while ([some telnet connection wants to do SSL])
{
ssl = SSL_new(ctx);
SSL_set_fd()
SSL_set_cipher_list(); SSL_set_connect_state();
SSL_connect();
do SSL_read(), SSL_write()
SSL_shutdown();
close FD;
SSL_free();
CRYPTO_cleanup_all_ex_data();
}
Any ideas would be appreciated.
Thanks,
David

______________________________________________________________________
OpenSSL Project http://www.openssl.org

______________________________________________________________________
OpenSSL Project http://www.openssl.org
User Support Mailing List openssl-***@openssl.org
Automated List Manager ***@openssl.org

[Dr. Stephen Henson]

Post by David
Hi,
I'm using tn3270 sessions running over SSL. I may have up to 124 sessions activated concurrently, although I plan to get up to 250 sessions at some point.
Whenever the sessions are stopped and restarted, I notice intermittently that memory grows in multiples of 4K bytes.
I'm running on AIX 5.1, 5.2 and 5.3 and using openssl-0.9.8l.
There doesn't appear to be an obvious memory leak in either my application or the OpenSSL stuff (all memory allocated when the sessions are started are freed when the sessions are stopped).
SSL_library_init();
meth = TLSv1_client_method();
RAND_seed();
ctx = SSL_CTX_new(meth);
while ([some telnet connection wants to do SSL])
{
ssl = SSL_new(ctx);
SSL_set_fd()
SSL_set_cipher_list();
SSL_set_connect_state();
SSL_connect();
do SSL_read(), SSL_write()
SSL_shutdown();
close FD;
SSL_free();
CRYPTO_cleanup_all_ex_data();
}
Any ideas would be appreciated.
Thanks,
David

Some cleanups occur on each connection and others only when the application
shuts down.

You should *not* call CRYPTO_cleanup_all_ex_data() on every SSL connection
because later SSL connections may use it and end up not freeing data
correctly.

This is especially an issue if connections use compression (OpenSSL compiled
against zlib) as it is by default in some linux distributions.

Steve.
--
Dr Stephen N. Henson. OpenSSL project core developer.
Commercial tech support now available see: http://www.openssl.org
______________________________________________________________________
OpenSSL Project http://www.openssl.org
User Support Mailing List openssl-***@openssl.org
Automated List Manager ***@openssl.org

[David]

Thanks. Are there any others which are mentioned under
http://www.openssl.org/support/faq.html#PROG13 which I should be wary of
for similar reasons?

David

--------------------------------------------------
From: "Dr. Stephen Henson" <***@openssl.org>
Sent: Friday, January 08, 2010 10:55 AM
To: <openssl-***@openssl.org>
Subject: Re: memory growing when using SSL connections

Post by Dr. Stephen Henson

Post by David
Hi,
I'm using tn3270 sessions running over SSL. I may have up to 124 sessions
activated concurrently, although I plan to get up to 250 sessions at some
point.
Whenever the sessions are stopped and restarted, I notice intermittently
that memory grows in multiples of 4K bytes.
I'm running on AIX 5.1, 5.2 and 5.3 and using openssl-0.9.8l.
There doesn't appear to be an obvious memory leak in either my
application or the OpenSSL stuff (all memory allocated when the sessions
are started are freed when the sessions are stopped).
SSL_library_init();
meth = TLSv1_client_method();
RAND_seed();
ctx = SSL_CTX_new(meth);
while ([some telnet connection wants to do SSL])
{
ssl = SSL_new(ctx);
SSL_set_fd()
SSL_set_cipher_list();
SSL_set_connect_state();
SSL_connect();
do SSL_read(), SSL_write()
SSL_shutdown();
close FD;
SSL_free();
CRYPTO_cleanup_all_ex_data();
}
Any ideas would be appreciated.
Thanks,
David

Some cleanups occur on each connection and others only when the
application
shuts down.
You should *not* call CRYPTO_cleanup_all_ex_data() on every SSL connection
because later SSL connections may use it and end up not freeing data
correctly.
This is especially an issue if connections use compression (OpenSSL
compiled
against zlib) as it is by default in some linux distributions.
Steve.
--
Dr Stephen N. Henson. OpenSSL project core developer.
Commercial tech support now available see: http://www.openssl.org
______________________________________________________________________
OpenSSL Project http://www.openssl.org

______________________________________________________________________
OpenSSL Project http://www.openssl.org
User Support Mailing List openssl-***@openssl.org
Automated List Manager ***@openssl.org

[Dr. Stephen Henson]

Post by David
Thanks. Are there any others which are mentioned under
http://www.openssl.org/support/faq.html#PROG13 which I should be wary of
for similar reasons?

Yes all of those should really only be called once on application exit. As it
says: "These internal tables can be freed up when an application closes using
various functions".

Some of the functions will restart later but others (due to current API
limitations) will not. In other cases there may be undesirable side effects:
for example if a crypto device goes through a complex startup procedure when
it is initialised.

Steve.
--
Dr Stephen N. Henson. OpenSSL project core developer.
Commercial tech support now available see: http://www.openssl.org
______________________________________________________________________
OpenSSL Project http://www.openssl.org
User Support Mailing List openssl-***@openssl.org
Automated List Manager ***@openssl.org

[Jeremy Hunt]

Hi David et al,

On reading the responses so far two new thoughts occur to me:

1. In view of Dr Henson's response, I wonder if removing the
CRYPTO_cleanup_all_ex_data() call in your loop will fix the problem.
Perhaps reusing the context structure after calling it may have the
reverse effect.

2. It may not be an SSL problem at all. Can you remove the SSL calls
from your application and see if you still get the memory leak? Your
underlying telnet application may be the cause.

Good Luck,

Jeremy

[safeTgram (safetgram-in) receive status: NOT encrypted, NOT signed.]

Post by David
Hi,
I'm using tn3270 sessions running over SSL. I may have up to 124 sessions activated concurrently, although I plan to get up to 250 sessions at some point.
Whenever the sessions are stopped and restarted, I notice intermittently that memory grows in multiples of 4K bytes.
I'm running on AIX 5.1, 5.2 and 5.3 and using openssl-0.9.8l.
There doesn't appear to be an obvious memory leak in either my application or the OpenSSL stuff (all memory allocated when the sessions are started are freed when the sessions are stopped).
SSL_library_init();
meth = TLSv1_client_method();
RAND_seed();
ctx = SSL_CTX_new(meth);
while ([some telnet connection wants to do SSL])
{
ssl = SSL_new(ctx);
SSL_set_fd()
SSL_set_cipher_list();
SSL_set_connect_state();
SSL_connect();
do SSL_read(), SSL_write()
SSL_shutdown();
close FD;
SSL_free();
CRYPTO_cleanup_all_ex_data();
}
Any ideas would be appreciated.
Thanks,
David

Some cleanups occur on each connection and others only when the application
shuts down.
You should *not* call CRYPTO_cleanup_all_ex_data() on every SSL connection
because later SSL connections may use it and end up not freeing data
correctly.
This is especially an issue if connections use compression (OpenSSL compiled
against zlib) as it is by default in some linux distributions.
Steve.
--
Dr Stephen N. Henson. OpenSSL project core developer.
Commercial tech support now available see: http://www.openssl.org
______________________________________________________________________
OpenSSL Project http://www.openssl.org

[David]

Hi Jeremy,

I did try removing the CRYPTO_cleanup_all_ex_data() call, based on Dr.
Henson's response, but I still have the same problem.

I bypassed the SSL calls and used pure telnet to my server and there
were no signs of the application growing.

Incidentally, I don't see this problem when running on a Solaris box.
Perhaps, it has something to do with the AIX environment.

Regards,
David

--------------------------------------------------
From: "Jeremy Hunt" <***@optimation.com.au>
Sent: Sunday, January 10, 2010 6:56 PM
To: <openssl-***@openssl.org>
Subject: Re: memory growing when using SSL connections

Post by Jeremy Hunt
Hi David et al,
1. In view of Dr Henson's response, I wonder if removing the
CRYPTO_cleanup_all_ex_data() call in your loop will fix the problem.
Perhaps reusing the context structure after calling it may have the
reverse effect.
2. It may not be an SSL problem at all. Can you remove the SSL calls from
your application and see if you still get the memory leak? Your underlying
telnet application may be the cause.
Good Luck,
Jeremy

[safeTgram (safetgram-in) receive status: NOT encrypted, NOT signed.]

Post by David
Hi,
I'm using tn3270 sessions running over SSL. I may have up to 124
sessions activated concurrently, although I plan to get up to 250
sessions at some point.
Whenever the sessions are stopped and restarted, I notice intermittently
that memory grows in multiples of 4K bytes.
I'm running on AIX 5.1, 5.2 and 5.3 and using openssl-0.9.8l. There
doesn't appear to be an obvious memory leak in either my application or
the OpenSSL stuff (all memory allocated when the sessions are started
are freed when the sessions are stopped).
SSL_library_init();
meth = TLSv1_client_method();
RAND_seed();
ctx = SSL_CTX_new(meth);
while ([some telnet connection wants to do SSL])
{
ssl = SSL_new(ctx);
SSL_set_fd()
SSL_set_cipher_list(); SSL_set_connect_state();
SSL_connect();
do SSL_read(), SSL_write()
SSL_shutdown();
close FD;
SSL_free();
CRYPTO_cleanup_all_ex_data();
}
Any ideas would be appreciated. Thanks,
David

Some cleanups occur on each connection and others only when the
application
shuts down.
You should *not* call CRYPTO_cleanup_all_ex_data() on every SSL
connection
because later SSL connections may use it and end up not freeing data
correctly.
This is especially an issue if connections use compression (OpenSSL
compiled
against zlib) as it is by default in some linux distributions.
Steve.
--
Dr Stephen N. Henson. OpenSSL project core developer.
Commercial tech support now available see: http://www.openssl.org
______________________________________________________________________
OpenSSL Project http://www.openssl.org

--
"The most likely way for the world to be destroyed, most experts agree, is
by accident. That's where we come in; we're computer professionals. We
cause accidents." -- Nathaniel Borenstein, co-creator of MIME
______________________________________________________________________
OpenSSL Project http://www.openssl.org

______________________________________________________________________
OpenSSL Project http://www.openssl.org
User Support Mailing List openssl-***@openssl.org
Automated List Manager ***@openssl.org

[Kevin Regan]

I had a similar issue that was fixed by moving the CRYPTO_cleanup_all_ex_data command to the end of the application (as has been suggested). Have you tried running Valgrind with the application? This may tell you where the memory leak is occurring.

--Kevin

p.s. You will need to compile OpenSSL with the -DPURIFY option (and you may need to grab a recent patch that I mailed to openssl-dev to get -DPURIFY working properly).

--Kevin

-----Original Message-----
From: owner-openssl-***@openssl.org [mailto:owner-openssl-***@openssl.org] On Behalf Of David
Sent: Monday, January 11, 2010 11:48 AM
To: openssl-***@openssl.org
Subject: Re: memory growing when using SSL connections

Hi Jeremy,

I did try removing the CRYPTO_cleanup_all_ex_data() call, based on Dr.
Henson's response, but I still have the same problem.

I bypassed the SSL calls and used pure telnet to my server and there
were no signs of the application growing.

Incidentally, I don't see this problem when running on a Solaris box.
Perhaps, it has something to do with the AIX environment.

Regards,
David

--------------------------------------------------
From: "Jeremy Hunt" <***@optimation.com.au>
Sent: Sunday, January 10, 2010 6:56 PM
To: <openssl-***@openssl.org>
Subject: Re: memory growing when using SSL connections

Post by Jeremy Hunt
Hi David et al,
1. In view of Dr Henson's response, I wonder if removing the
CRYPTO_cleanup_all_ex_data() call in your loop will fix the problem.
Perhaps reusing the context structure after calling it may have the
reverse effect.
2. It may not be an SSL problem at all. Can you remove the SSL calls from
your application and see if you still get the memory leak? Your underlying
telnet application may be the cause.
Good Luck,
Jeremy

[safeTgram (safetgram-in) receive status: NOT encrypted, NOT signed.]

Post by David
Hi,
I'm using tn3270 sessions running over SSL. I may have up to 124
sessions activated concurrently, although I plan to get up to 250
sessions at some point.
Whenever the sessions are stopped and restarted, I notice intermittently
that memory grows in multiples of 4K bytes.
I'm running on AIX 5.1, 5.2 and 5.3 and using openssl-0.9.8l. There
doesn't appear to be an obvious memory leak in either my application or
the OpenSSL stuff (all memory allocated when the sessions are started
are freed when the sessions are stopped).
SSL_library_init();
meth = TLSv1_client_method();
RAND_seed();
ctx = SSL_CTX_new(meth);
while ([some telnet connection wants to do SSL])
{
ssl = SSL_new(ctx);
SSL_set_fd()
SSL_set_cipher_list(); SSL_set_connect_state();
SSL_connect();
do SSL_read(), SSL_write()
SSL_shutdown();
close FD;
SSL_free();
CRYPTO_cleanup_all_ex_data();
}
Any ideas would be appreciated. Thanks,
David

Some cleanups occur on each connection and others only when the
application
shuts down.
You should *not* call CRYPTO_cleanup_all_ex_data() on every SSL
connection
because later SSL connections may use it and end up not freeing data
correctly.
This is especially an issue if connections use compression (OpenSSL
compiled
against zlib) as it is by default in some linux distributions.
Steve.
--
Dr Stephen N. Henson. OpenSSL project core developer.
Commercial tech support now available see: http://www.openssl.org
______________________________________________________________________
OpenSSL Project http://www.openssl.org

--
"The most likely way for the world to be destroyed, most experts agree, is
by accident. That's where we come in; we're computer professionals. We
cause accidents." -- Nathaniel Borenstein, co-creator of MIME
______________________________________________________________________
OpenSSL Project http://www.openssl.org

______________________________________________________________________
OpenSSL Project http://www.openssl.org
User Support Mailing List openssl-***@openssl.org
Automated List Manager ***@openssl.org
______________________________________________________________________
OpenSSL Project http://www.openssl.org
User Support Mailing List openssl-***@openssl.org
Automated List Manager ***@openssl.org

[David]

Kevin,

Valgrind apparently is a Linux based tool, but I'm having this problem on
AIX. I actually have a simple memory leak finder that showed that SSL did
not have an obvious memory leak. It looks like most, if not all memory
management functions for SSL goes thru CRYPTO_malloc(), CRYPTO_realloc() and
CRYPTO_free() and their variants. I found a couple of calloc()s elsewhere,
but it looks like everything else got covered in these functions.

So pretty much, when memory is allocated (or reallocated) I put it on a
linked list. When it is freed, I remove it from the list. After all my
sessions have stopped, I see that my linked list hasn't grown from the
previous run.

I'm a bit suspicious about the realloc()s, since this is can be a source of
memory fragmentation. However, what is curious is that I don't see this
problem on Solaris.
IBM has a product called PURIFYPLUS for detecting memory leaks. Perhaps I'll
see if I can run it with the application and see if it shows up anything.

Thanks,
David

--------------------------------------------------
From: "Kevin Regan" <***@f5.com>
Sent: Monday, January 11, 2010 3:19 PM
To: <openssl-***@openssl.org>
Subject: RE: memory growing when using SSL connections

Post by Kevin Regan
I had a similar issue that was fixed by moving the
CRYPTO_cleanup_all_ex_data command to the end of the application (as has
been suggested). Have you tried running Valgrind with the application?
This may tell you where the memory leak is occurring.
--Kevin
p.s. You will need to compile OpenSSL with the -DPURIFY option (and you
may need to grab a recent patch that I mailed to openssl-dev to
get -DPURIFY working properly).
--Kevin
-----Original Message-----
Sent: Monday, January 11, 2010 11:48 AM
Subject: Re: memory growing when using SSL connections
Hi Jeremy,
I did try removing the CRYPTO_cleanup_all_ex_data() call, based on Dr.
Henson's response, but I still have the same problem.
I bypassed the SSL calls and used pure telnet to my server and there
were no signs of the application growing.
Incidentally, I don't see this problem when running on a Solaris box.
Perhaps, it has something to do with the AIX environment.
Regards,
David
--------------------------------------------------
Sent: Sunday, January 10, 2010 6:56 PM
Subject: Re: memory growing when using SSL connections

Post by Jeremy Hunt
Hi David et al,
1. In view of Dr Henson's response, I wonder if removing the
CRYPTO_cleanup_all_ex_data() call in your loop will fix the problem.
Perhaps reusing the context structure after calling it may have the
reverse effect.
2. It may not be an SSL problem at all. Can you remove the SSL calls from
your application and see if you still get the memory leak? Your
underlying
telnet application may be the cause.
Good Luck,
Jeremy

[safeTgram (safetgram-in) receive status: NOT encrypted, NOT signed.]

Post by David
Hi,
I'm using tn3270 sessions running over SSL. I may have up to 124
sessions activated concurrently, although I plan to get up to 250
sessions at some point.
Whenever the sessions are stopped and restarted, I notice
intermittently
that memory grows in multiples of 4K bytes.
I'm running on AIX 5.1, 5.2 and 5.3 and using openssl-0.9.8l. There
doesn't appear to be an obvious memory leak in either my application or
the OpenSSL stuff (all memory allocated when the sessions are started
are freed when the sessions are stopped).
SSL_library_init();
meth = TLSv1_client_method();
RAND_seed();
ctx = SSL_CTX_new(meth);
while ([some telnet connection wants to do SSL])
{
ssl = SSL_new(ctx);
SSL_set_fd()
SSL_set_cipher_list(); SSL_set_connect_state();
SSL_connect();
do SSL_read(), SSL_write()
SSL_shutdown();
close FD;
SSL_free();
CRYPTO_cleanup_all_ex_data();
}
Any ideas would be appreciated. Thanks,
David

Some cleanups occur on each connection and others only when the
application
shuts down.
You should *not* call CRYPTO_cleanup_all_ex_data() on every SSL
connection
because later SSL connections may use it and end up not freeing data
correctly.
This is especially an issue if connections use compression (OpenSSL
compiled
against zlib) as it is by default in some linux distributions.
Steve.
--
Dr Stephen N. Henson. OpenSSL project core developer.
Commercial tech support now available see: http://www.openssl.org
______________________________________________________________________
OpenSSL Project http://www.openssl.org

--
"The most likely way for the world to be destroyed, most experts agree,
is
by accident. That's where we come in; we're computer professionals. We
cause accidents." -- Nathaniel Borenstein, co-creator of MIME
______________________________________________________________________
OpenSSL Project http://www.openssl.org

______________________________________________________________________
OpenSSL Project http://www.openssl.org
______________________________________________________________________
OpenSSL Project http://www.openssl.org

______________________________________________________________________
OpenSSL Project http://www.openssl.org
User Support Mailing List openssl-***@openssl.org
Automated List Manager ***@openssl.org

[Dr. Stephen Henson]

Post by David
Kevin,
Valgrind apparently is a Linux based tool, but I'm having this problem on
AIX. I actually have a simple memory leak finder that showed that SSL did
not have an obvious memory leak. It looks like most, if not all memory
management functions for SSL goes thru CRYPTO_malloc(), CRYPTO_realloc()
and CRYPTO_free() and their variants. I found a couple of calloc()s
elsewhere, but it looks like everything else got covered in these
functions.
So pretty much, when memory is allocated (or reallocated) I put it on a
linked list. When it is freed, I remove it from the list. After all my
sessions have stopped, I see that my linked list hasn't grown from the
previous run.
I'm a bit suspicious about the realloc()s, since this is can be a source of
memory fragmentation. However, what is curious is that I don't see this
problem on Solaris.
IBM has a product called PURIFYPLUS for detecting memory leaks. Perhaps
I'll see if I can run it with the application and see if it shows up
anything.

OpenSSL has some built in leak detection. If you call something like:

MemCheck_start();

before a leaking section then:

CRYPTO_mem_leaks_fp(stderr);

after you think you've freed everything up you should get some useful results
if the leak is internal to OpenSSL. The output takes a bit of getting used to
(I usually run it twice setting breakpoints) but can be very useful.

Steve.
--
Dr Stephen N. Henson. OpenSSL project core developer.
Commercial tech support now available see: http://www.openssl.org
______________________________________________________________________
OpenSSL Project http://www.openssl.org
User Support Mailing List openssl-***@openssl.org
Automated List Manager ***@openssl.org

[David]

After running my app for a couple of iterations for over an hour, I get the
following:

[11:45:42] 10 file=stack.c, line=125, thread=15418, number=20,
address=20077768
[11:45:47] 8711 file=lhash.c, line=193, thread=15418, number=12,
address=20093E38
[11:45:42] 5 file=stack.c, line=125, thread=15418, number=20,
address=20077508
[11:45:47] 8180 file=stack.c, line=127, thread=15418, number=16,
address=20089848
[11:45:47] 8709 file=stack.c, line=125, thread=15418, number=20,
address=20093B48
[11:45:42] 23 file=lhash.c, line=193, thread=15418, number=12,
address=20077C78
[11:45:42] 35 file=ex_data.c, line=304, thread=15418, number=12,
address=20088BE8
[11:45:42] 12 file=lhash.c, line=193, thread=15418, number=12,
address=20077848
[11:45:47] 8181 file=lhash.c, line=193, thread=15418, number=12,
address=200898B8
[11:45:42] 7 file=lhash.c, line=193, thread=15418, number=12,
address=200775E8
[11:45:42] 36 file=stack.c, line=125, thread=15418, number=20,
address=20088C58
[11:45:47] 8710 file=stack.c, line=127, thread=15418, number=16,
address=20093BB8
[11:45:42] 37 file=stack.c, line=127, thread=15418, number=16,
address=20088CC8
[11:45:42] 22 file=lhash.c, line=121, thread=15418, number=64,
address=20077D58

thread=15418, file=err_def.c, line=346, info="int_thread_get (err.c)"

[11:45:42] 9 file=ex_data.c, line=304, thread=15418, number=12,
address=200776F8
[11:45:47] 8178 file=ex_data.c, line=304, thread=15418, number=12,
address=20089768
[11:45:47] 8708 file=ex_data.c, line=304, thread=15418, number=12,
address=200938C8
[11:45:42] 20 file=err_def.c, line=629, thread=15418, number=396,
address=200779F8
[11:45:42] 38 file=lhash.c, line=193, thread=15418, number=12,
address=20088D38
[11:45:42] 4 file=ex_data.c, line=304, thread=15418, number=12,
address=20077498
[11:45:42] 21 file=lhash.c, line=119, thread=15418, number=96,
address=20077C98

thread=15418, file=err_def.c, line=346, info="int_thread_get (err.c)"

[11:45:47] 8179 file=stack.c, line=125, thread=15418, number=20,
address=200897D8
[11:45:42] 11 file=stack.c, line=127, thread=15418, number=16,
address=200777D8
[11:45:42] 6 file=stack.c, line=127, thread=15418, number=16,
address=20077578
868 bytes leaked in 24 chunks

However, I get the same info over and over again (same time stamps), no
matter the amount of iterations. I believe this shows that
there is no continuous memory leak in SSL. The 868 bytes reported as a leak
by CRYPTO_mem_leaks_fp() appears to be some initial memory
that has been allocated for the life of the application.

Thanks,
David
--------------------------------------------------
From: "Dr. Stephen Henson" <***@openssl.org>
Sent: Monday, January 11, 2010 7:17 PM
To: <openssl-***@openssl.org>
Subject: Re: memory growing when using SSL connections

Post by David
Kevin,
Valgrind apparently is a Linux based tool, but I'm having this problem on
AIX. I actually have a simple memory leak finder that showed that SSL did
not have an obvious memory leak. It looks like most, if not all memory
management functions for SSL goes thru CRYPTO_malloc(), CRYPTO_realloc()
and CRYPTO_free() and their variants. I found a couple of calloc()s
elsewhere, but it looks like everything else got covered in these
functions.
So pretty much, when memory is allocated (or reallocated) I put it on a
linked list. When it is freed, I remove it from the list. After all my
sessions have stopped, I see that my linked list hasn't grown from the
previous run.
I'm a bit suspicious about the realloc()s, since this is can be a source
of
memory fragmentation. However, what is curious is that I don't see this
problem on Solaris.
IBM has a product called PURIFYPLUS for detecting memory leaks. Perhaps
I'll see if I can run it with the application and see if it shows up
anything.

MemCheck_start();
CRYPTO_mem_leaks_fp(stderr);
after you think you've freed everything up you should get some useful
results
if the leak is internal to OpenSSL. The output takes a bit of getting used
to
(I usually run it twice setting breakpoints) but can be very useful.
Steve.
--
Dr Stephen N. Henson. OpenSSL project core developer.
Commercial tech support now available see: http://www.openssl.org
______________________________________________________________________
OpenSSL Project http://www.openssl.org

______________________________________________________________________
OpenSSL Project http://www.openssl.org
User Support Mailing List openssl-***@openssl.org
Automated List Manager ***@openssl.org

[David]

Jeremy,

The application still grows intermittently by multiples of 4KB although
the memory leak tools provided by SSL and myself shows no memory leaks.

The start for the checking of the memory leaks is before the SSL_new()
and the actual check is after SSL_free().

It should be noted that on AIX, the process grows quickly initially and
then slows down after many iterations and hours, but I need to verify if it
actually comes to a halt.

On the Solaris platform, the memory size of the process is bigger, but
it stops growing almost immediately (after 2 iterations of 124 sessions
stopped and restarted)

On the AIX platform, it doesn't get that big as quickly, but keeps on
growing slowly after many iterations. The pagesize on Solaris is 8192 bytes
whereas the pagesize on AIX 4096 bytes which may
explain the diff in the memory sizes.

Perhaps, as you say, I should use a more general tool for checking this
stuff.

Thanks,
David

--------------------------------------------------
From: "Jeremy Hunt" <***@optimation.com.au>
Sent: Tuesday, January 12, 2010 6:14 PM
To: <***@bellsouth.net>; <openssl-***@openssl.org>
Subject: Re: memory growing when using SSL connections

[safeTgram (safetgram-in) receive status: NOT encrypted, NOT signed.]
After running my app for a couple of iterations for over an hour, I get
[11:45:42] 10 file=stack.c, line=125, thread=15418, number=20,
address=20077768

...

868 bytes leaked in 24 chunks
However, I get the same info over and over again (same time stamps), no
matter the amount of iterations. I believe this shows that
there is no continuous memory leak in SSL. The 868 bytes reported as a
leak by CRYPTO_mem_leaks_fp() appears to be some initial memory
that has been allocated for the life of the application.
Thanks,
David

David, I am a bit confused by what you say ..
Initially you said the application was growing intermittently in 4KB
multiples. Do you see this behaviour when you add Dr Henson's calls?
If so then where have you added the calls? I presume inside the loop after
SSL_new() and SSL_free(), if all you did previously was remove
CRYPTO_cleanup_all_ex_data().
If both of these statements are correct then either CRYPTO_mem_leaks_fp()
missed the leak or it really isn't in your SSL code. Yet you say that
without the SSL code you get no memory leaks, ... and you get no memory
leaks on the Solaris platform. Looks like you may need a more general
tool. Valgrind 3.3 does work on some versions of AIX on some hardware
(experimentally, apparently), you could try to find a copy of the source,
approach the developers or try to build the latest version on AIX. And
there is purify or insure++.
How large does the memory leak manifest itself? Are there any broken
connections hanging around that might keep memory until they timeout?
There isn't some AIX kernel tuning parameter that keeps memory that has
been freed reported as allocated until another process needs it is there,
in other words tuning for a delayed memory cleanup algorithm?

______________________________________________________________________
OpenSSL Project http://www.openssl.org
User Support Mailing List openssl-***@openssl.org
Automated List Manager ***@openssl.org

[Jeremy Hunt]

[safeTgram (safetgram-in) receive status: NOT encrypted, NOT signed.]
After running my app for a couple of iterations for over an hour, I get the
[11:45:42] 10 file=stack.c, line=125, thread=15418, number=20,
address=20077768

...

868 bytes leaked in 24 chunks
However, I get the same info over and over again (same time stamps), no
matter the amount of iterations. I believe this shows that
there is no continuous memory leak in SSL. The 868 bytes reported as a leak
by CRYPTO_mem_leaks_fp() appears to be some initial memory
that has been allocated for the life of the application.
Thanks,
David

David, I am a bit confused by what you say ..

Initially you said the application was growing intermittently in 4KB
multiples. Do you see this behaviour when you add Dr Henson's calls?

If so then where have you added the calls? I presume inside the loop
after SSL_new() and SSL_free(), if all you did previously was remove
CRYPTO_cleanup_all_ex_data().

If both of these statements are correct then either
CRYPTO_mem_leaks_fp() missed the leak or it really isn't in your SSL
code. Yet you say that without the SSL code you get no memory leaks, ...
and you get no memory leaks on the Solaris platform. Looks like you may
need a more general tool. Valgrind 3.3 does work on some versions of AIX
on some hardware (experimentally, apparently), you could try to find a
copy of the source, approach the developers or try to build the latest
version on AIX. And there is purify or insure++.

Some other thoughts that you may have already considered:
How large does the memory leak manifest itself? Are there any broken
connections hanging around that might keep memory until they timeout?
There isn't some AIX kernel tuning parameter that keeps memory that has
been freed reported as allocated until another process needs it is
there, in other words tuning for a delayed memory cleanup algorithm?

______________________________________________________________________
OpenSSL Project http://www.openssl.org
User Support Mailing List openssl-***@openssl.org
Automated List Manager ***@openssl.org