script hangs on the removeWords step

[maraaaa]

The script seems to get hung on the line:

corpus <- tm_map(corpus, removeWords, stopwords('english'))

I'm using the largest Digital Ocean droplet possible, and when I get to this point R pegs loadavg at 1, so it appears to be leveraging all 20 cores, however, even running against just two days worth of logs this step never completes.

If I skip that step in the script, I get word clouds like: http://i.imgur.com/WSo1VJ2.png

Has anyone else run into this?

> library(jsonlite)
> library(tm)
> library(wordcloud)
Loading required package: RColorBrewer
> sessionInfo()
R version 3.3.0 (2016-05-03)
Platform: x86_64-pc-linux-gnu (64-bit)
Running under: Ubuntu 14.04.4 LTS

locale:
 [1] LC_CTYPE=en_US.UTF-8       LC_NUMERIC=C
 [3] LC_TIME=en_US.UTF-8        LC_COLLATE=en_US.UTF-8
 [5] LC_MONETARY=en_US.UTF-8    LC_MESSAGES=en_US.UTF-8
 [7] LC_PAPER=en_US.UTF-8       LC_NAME=C
 [9] LC_ADDRESS=C               LC_TELEPHONE=C
[11] LC_MEASUREMENT=en_US.UTF-8 LC_IDENTIFICATION=C

attached base packages:
[1] stats     graphics  grDevices utils     datasets  methods   base

other attached packages:
[1] wordcloud_2.5      RColorBrewer_1.1-2 tm_0.5             jsonlite_0.9.19

loaded via a namespace (and not attached):
[1] Rcpp_0.11.0 slam_0.1-34

Thanks!

[justinxreese]

We haven't run this in a while, I'm wondering if Slack changed what the export looks like. Could you see what the corpus is at that point?

All that line does is remove stopwords so it's surprising that it would have any impact on whether or not dates were in the corpus (also surprising that it causes the program to hang with that many resources, but that's harder to debug).

[maraaaa]

So I did a:

> print(corpus)
A corpus with 55581 text documents
> sprintf("%s", corpus)

The sprintf dumps a bunch of lines like:

     [2585] "meeting from "  



     [2586] "so no leaving early for the holiday who does that" 

[maraaaa]

Interestingly, I tried this on a VMware backed CentOS 6.5 VM. I did have to bump the ram to 4GB to be able to run this script, but it runs really fast. Also, I see R spawn additional processes, even when the VM was a single core. That did not happen on the Ubuntu VM, there was only ever one R process.

My Word cloud: http://i.imgur.com/rOiCDSH.png

I was wondering if R didn't recognize the additional cores, because I had initially tried this on the smallest droplet... but CentOS spun child processes even with only one core, and spun off up to 3 or 4 when I bumped the number of cores to 4 on the CentOS VM.

I'm kinda curious now to try it on a CentOS Droplet.

I did have to install TM v5, I used the most current version on the CentOS VM, so maybe it's worth trying the 0.5.0 version of TM on CentOS too...

[alexcurtin]

I am running the same version of tm and R and mine is hanging on the same

 # Remove some stopwords
 > corpus <- tm_map(corpus, removeWords, stopwords('english'))

Any ideas?