So does anyone else's key fob quit working when the fob itself is a little cold?

I noticed this in the fall when I would play golf and throw my keys in the golf cart, play a quick nine holes, and when I get back to the car, the key fob won't work. It works fine again once its been in my pants pocket long enough to warm it up again, but it seems like if my key fob is any cooler than like 55 degrees, it doesn't work. Now that its January, I can't even keep my keys in my jacket pocket without it failing on me. Anyone have any ideas on what is going on?

Also, its probably worth mentioning that my car does not have the key-less ignition option.