Re: Problem with redis on AWS and transition to dynamodb

Well, my money are on AWS Elasticache, see if bypassing their proxy will do any good. It's probably a good idea to do so in testing environment with high load targeting elasticache and nothing else. Also it might be a good idea to write to AWS too, maybe there're other people with the same problem out there.

On 6 Dec 2016, at 18:03, Andreas Kuhne <andreas.kuhne@suitopia.com> wrote:

The way it is set up there are 2 cache nodes in a cluster - a primary and a secondary. Then I have a seperate address to connect to the cache cluster - so there probably is some sort of proxy there, but I can't manipulate it - it's in the AWS Elasticache offering. I don't want to bypass the proxy, because if the primary goes down, it won't automatically failover if I don't use it. I could however test to bypass it just to see if that changes anything.

Also I have setup the sessions backend via the PyPI package django-redis-sessions - so it handles the connections for me. I don't know if it reuses the connections or not - however looking at the monitoriing of the redis server, it doesn't have that many connections - I think the django-redis-sessions plugin has a short timeout value. 

However this has worked perfectly until just a month ago, so I don't really see why these changes have come now.

Med vänliga hälsningar,

Andréas Kühne

Software Development Manager

Suitopia Scandinavia AB

2016-12-06 15:46 GMT+01:00 GMail <roboslone@gmail.com>:

You didn't answer about proxy. Also, do you reuse connections to redis.example.com? Or create a new one for every request? If you aren't reusing them, maybe you should add timeout for connection creation and retry creating after timeout has exceeded. I know this doesn't solve the problem, but at least you wouldn't have to wait for server timeout.

I still think something is proxying your request to redis.example.com and this something is what gives you timeouts and connection errors.

On 6 Dec 2016, at 17:41, Andreas Kuhne <andreas.kuhne@suitopia.com> wrote:

Hi,

Thanks for you answer - I don't think the problem is with django either, but thought that maybe someone else has run into the same issues on AWS. My cache cluster worked fine for over 2 year as well, but now, not so much :-)

2016-12-06 14:35 GMT+01:00 GMail <roboslone@gmail.com>:

Hi!

Do you by any chance have any proxy on top of Redis? Seems like Django has nothing to do with it, though.

I don't know anything about dynamodb really, but I think implementing Django cache with dynamo as backend should work. I've implemented Redis Cluster cache for my project and it works fine. Here're the docs about session engines: https://docs.djangoproject.com/en/1.10/topics/http/sessions/#using-cached-sessions

On 6 Dec 2016, at 15:55, Andreas Kuhne <andreas.kuhne@suitopia.com> wrote:

Hi,

We are having a strange problem with our redis elasticache instances on AWS. We have our sessions stored in a redis cluster on AWS. Our webservers sometimes get a:

* Error connecting to redis.example.com:6379. timed out 
* Timeout reading from socket

(I haven't included our real domain for the redis servers :-))

Both of these are related to our webservers connecting to our redis server. This started about a month ago, and worked perfectly for nearly 2 years before that. I really don't understand why this is happening. I know that the webservers have access to the redis servers, so I really can't understand the problem. Has anyone else experienced a similar problem?

That was the first part of my question, my second is related to this though. For costsaving I would really like to dump the redis cluster and change to a dynamodb based session store. I have found a pypi package that works for the implementation, but when I tried it about 2 years ago, it failed miserably. We ended up having people get new sessions all the time and the sessions weren't being persisted corrrectly (each request ended up generating a new session).

Has anyone successfully implemented a session store on dynamodb? With 2 webservers behind an loadbalancer?

Also, has anyone any ideas of how to move the sessions from our redis store to the dynamodb database?

Any tips / tricks would be very appreciated.

Regards,

Andréas

--
You received this message because you are subscribed to the Google Groups "Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to django-users+unsubscribe@googlegroups.com.
To post to this group, send email to django-users@googlegroups.com.
Visit this group at https://groups.google.com/group/django-users.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-users/CALXYUb%3D89GTJ33FgE_DpjofdgOn2fYqVoctzo00NOq5K26Twmw%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to django-users+unsubscribe@googlegroups.com.
To post to this group, send email to django-users@googlegroups.com.
Visit this group at https://groups.google.com/group/django-users.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-users/13E0B717-29E6-43D5-AAD9-0FD4A4424186%40gmail.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to django-users+unsubscribe@googlegroups.com.
To post to this group, send email to django-users@googlegroups.com.
Visit this group at https://groups.google.com/group/django-users.

To view this discussion on the web visit https://groups.google.com/d/msgid/django-users/CALXYUbkum1Ou5x6yTqFxz-T1pG2%3D9_Km-_XbWBiFur34dVnXyg%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to django-users+unsubscribe@googlegroups.com.
To post to this group, send email to django-users@googlegroups.com.
Visit this group at https://groups.google.com/group/django-users.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-users/E58F1469-2A35-45DC-8F5D-7DC9D6E61650%40gmail.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to django-users+unsubscribe@googlegroups.com.
To post to this group, send email to django-users@googlegroups.com.
Visit this group at https://groups.google.com/group/django-users.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-users/CALXYUbm_zJa48VbEj%2BSevZQ%3DX%3DT-tR6%3DL%2Bf9Hby95Ahhpm2CHQ%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.