Lactose and Casein are two components found in milk and diary products. Lactose is sugar, while casein is a protein. People with lactose intolerance (more common in people of Asian and African-Caribbean descent) do not produce the enzyme lactase, needed to break down the sugar in the digestive system. Lactase normally stops being produced after infancy, however, in some populations where dairy has made up a large part of the diet, particularly Caucasians, the gene has evolved to produce lactase throughout adult life. Lactose intolerance is a genetic condition that can cause unpleasant symptoms such as bloating, gas, cramps and diarrhoea.
Some people, particularly babies, develop an allergy towards casein. This is NOT a genetic condition but usually a result of a compromised or underdeveloped digestive system. 
Luckily, there are plenty of dairy-free alternatives to choose from: