How did my teeth become so yellow?

Your teeth are made up of layers of material: a hard top coat of enamel and a softer, inner core of dentin. When you eat, drink or smoke, a third layer of all the stuff you eat or drink builds up over time on top of the hard enamel, forming a pellicle film. Sometimes excess fluoride or even tetracycline can cause staining as well. When you brush your teeth you are getting rid of these materials and the film, but some remain. If you use a whitening toothpaste, you are getting off more of the film. When you go to your dentist for your regular check up and cleaning, the abrasive brushing will get rid of some of this film as well.

Why is it so difficult to remove? Even though your enamel layer is hard, it is not impervious to the pellicle layer, and eventually some of the material is soaked into your enamel. It's not harmful to you but it does cause this discoloration you are anxious to remove.