8

I am considering using wget -p -k -n Tex.stackexchange and thus save all posts to some future emergency if I or the site goes offline. Do you think this is a good idea?

34

No, scraping the site like that is a quick way to "Too many requests from this IP address".

A data dump of all SE sites is available from Internet Archive, including

tex.stackexchange.com.7z    04-Mar-2019 16:44   484.3M 

This file includes questions, answers, comments, and more: see Database schema documentation for the public data dump and SEDE

  • 3
    Hey, at least compressed, we are larger than unix.se and even among the top five (superuser.com is with 815M the biggest, followed by askubuntu.com, serverfault.se, and, interestingly, ru.se). – Daniel Apr 25 at 8:52
  • 1
    Math.SE is over 1G. – user69318 Apr 25 at 10:47
  • Oh, I apparently missed that one. math.se being the biggest is nevertheless remarkable. – Daniel Apr 25 at 15:36
  • @Daniel superuser.com is the biggest? What about stackoverflow.com-Posts.7z 13.2G? (and that is not even all of it) – Henri Menke Apr 25 at 21:48
  • 1
    Wow, many sites even are smaller than stackoverflow.com-Votes.7z, which is 997.3M. Insane. – Just a student Apr 26 at 10:59
  • 2
    @Daniel I am not really sure what gets included in the dump, but if screenshots/images are included that may explain why TeX.SE is larger than others, given that a large amount of answers (but also questions) have images here. On Math.SE they use MathJax, but if that is included in the dump in rendered form (i.e., as images) then that also adds to the size of the archive. – Marijn Apr 26 at 13:25
  • 1
    @Marijn Images are not included in data dumps, they are hosted externally by Imgur. The questions and answers are contained in HTML format in the Posts table. The Markdown source is available in the PostHistory table. The link at the end of my answer has the details. – user69318 Apr 26 at 14:33

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .