By default, Ubuntu comes with a lot of non-english fonts (Japanese, Cyrillic etc.). There is nothing inherently wrong with this, but if you do all you work in english you have scroll through a bloated list useless fonts every time you want to do a font selection.

andrew@thisblog seems to have come up with a very quick fix. Uninstall them with single command:

sudo apt-get remove ttf-indic-fonts-core ttf-kacst-one ttf-khmeros-core ttf-lao ttf-punjabi-fonts ttf-takao-pgothic ttf-thai-tlwg ttf-unfonts-core ttf-wqy-microhei

Apparently, I am not the only one who has noticed:
Ubuntu Brainstorm: Idea #9275: Remove this chaos with a fonts!!!