Yes, tyres will degrade over time. Even if they are not used frequently.
As tyres age, their makeup becomes more susceptible to damage (i.e. you will start to see cracking and splitting on old tyres), which can compromise their performance.
This is especially hazardous when driving at high speeds or on wet roads.
For these reasons, it is important to inspect your tyres for signs of wear or damage on a regular basis. When you are checking air pressure is a good time to give all your tyres a visual inspection.
If you have had your tyres for more than 5 years, regardless of their tread depth, you should look to replace them.
Proper storage is also crucial in ensuring that tyres last for as long as possible. Where possible, keep them away from sources of heat and excessive levels of moisture - in particular deep standing water.