In the most general sense instability in any system can be described by the idea that one parameter describing the system grows without bound for a small change in another parameter.
For the purpose of soft matter we are mostly concerned with two more specific formulations of instabilities. The first one is originally derived from structural engineering of solid materials. A structure is unstable when a small change in load applied to it leads to a large deflection. This is due to a positive feedback loop. Above a critical stress, the deflection due to the applied stress itself increases the stress. This is a very general mechanism that lies at the heart of many instability phenomenon. For solid structures the instability is very often buckling.