It turns out the same problem happens when you install a lot of new fonts. XeLaTeX takes a while to "load" them or whatever it needs to do, but after you let it run for a few minutes it will successfully compile the document. It only happens the first time you compile a document, everything will work as expected from the second time on.
The same problem happens with the GIMP: if you install new fonts it takes some time to rebuild the font cache and in the while it becomes unresponsive. After that, everything works fine.
The question is now closed and deleted. It seems to me that the question itself may be off topic, but maybe the answer may be useful to someone. What should I do?