## Scatter plot for still lifes (R help needed)

For discussion directly related to ConwayLife.com, such as requesting changes to how the forums or wiki function.

### Scatter plot for still lifes (R help needed)

I was thinking it would be nice to have a scatter plot of (total) still life counts vs. populations in the Catagolue article on the LifeWiki (which is why I'm sticking this in "Website Discussion", BTW), akin to the upper plot on page 2 of the full results for Andrzej Okrasinski's census.

That sort of thing's best done programmatically, so I turned to R, which I vaguely recalled being told was useful; but I've never used that before myself, and found it rather unwieldy, so I was wondering if anyone here either a) was an R wizard who could help improve the following plot, or b) had suggestions for a better tool.

For now here's the current best attempt at a plot I have:

Here's the R script used:

library(bit64)options(scipen=100)stilllifes <- read.table("xs_scatterplot.dat", header=TRUE);attach(stilllifes)## svg("xs_scatterplot.svg", width=10, height=10, antialias="subpixel")png("xs_scatterplot.png", width=640, height=640, antialias="cleartype")plot(Population, Count, log="y", lab=c(length(Population), ceiling(log10(max(Count)))+1, 7), xaxs="i")## text(Population, Count, Count, cex=0.6, pos=4, col="red")legend("topright", inset=0.05, title="Total still lifes", c("Count", "Linear fit", expression(paste("LOWESS fit, ", alpha, "=0.3"))), pch=c(21, -1, -1), lty=c(0, 1, 1), col=c("black", "black", "blue"))title(main="Total still lifes by population")abline(lm(log10(Count)~Population))lines(lowess(stilllifes, f=0.3), col="blue")dev.off()

And finally, the data file (note that this was generated from somewhat older data; I'd use newer data when generating an image to upload to the LifeWiki):

Population   Count4   372306632447005   52176679711956   228114390596397   60280585129998   11510834897599   613060516810   417980070811   123655693712   18649026077813   15767052614   7143998153715   42307417616   267230714217   14066517118   50277163819   1764047420   6287020821   146083322   928393023   45412624   40952925   20947926   14223127   3076128   293226029   898830   4391131   1713532   705233   17734   30035   2336   5137   938   40239   440   509241   242   6445   146   1

So, yeah. Any tips on how to improve this plot? R seems to offer precious little control over formatting, axes, labels, ticks etc., but maybe I'm just looking in the wrong places.

Alternatively, any other (better) tools to generate such plots? Programs that can be used non-interactively (i.e. called with a script as part of a shell script) would be very strongly preferred.
If you speak, your speech must be better than your silence would have been. — Arabian proverb

Catagolue: Apple Bottom • Life Wiki: Apple Bottom • Twitter: @_AppleBottom_

Proud member of the Pattern Raiders!

Apple Bottom

Posts: 1023
Joined: July 27th, 2015, 2:06 pm

### Re: Scatter plot for still lifes (R help needed)

Python + matplotlib?
Disclaimer: I have only heard of it.
Best wishes to you, Scorbie

Scorbie

Posts: 1376
Joined: December 7th, 2013, 1:05 am

### Re: Scatter plot for still lifes (R help needed)

Scorbie wrote:Python + matplotlib?
Disclaimer: I have only heard of it.

"matplotlib tries to make easy things easy and hard things possible" -- that certainly sounds promising. (Also, they're taking more than a subtle cue from Perl there, aren't they.)

I'll look into it, thanks!

EDIT: not much luck so far. Cygwin doesn't provide a package, and the pip package can't even be installed:

$pip install matplotlib[...] running build_ext building 'matplotlib.ft2font' extension Traceback (most recent call last): File "<string>", line 1, in <module> File "/tmp/pip-build-gwez0e7l/matplotlib/setup.py", line 277, in <module> **extra_args File "/usr/lib/python3.4/distutils/core.py", line 148, in setup dist.run_commands() File "/usr/lib/python3.4/distutils/dist.py", line 955, in run_commands self.run_command(cmd) File "/usr/lib/python3.4/distutils/dist.py", line 974, in run_command cmd_obj.run() File "/usr/lib/python3.4/site-packages/setuptools/command/install.py", line 61, in run return orig.install.run(self) File "/usr/lib/python3.4/distutils/command/install.py", line 539, in run self.run_command('build') File "/usr/lib/python3.4/distutils/cmd.py", line 313, in run_command self.distribution.run_command(command) File "/usr/lib/python3.4/distutils/dist.py", line 974, in run_command cmd_obj.run() File "/usr/lib/python3.4/distutils/command/build.py", line 126, in run self.run_command(cmd_name) File "/usr/lib/python3.4/distutils/cmd.py", line 313, in run_command self.distribution.run_command(command) File "/usr/lib/python3.4/distutils/dist.py", line 974, in run_command cmd_obj.run() File "/usr/lib/python3.4/site-packages/setuptools/command/build_ext.py", line 50, in run _build_ext.run(self) File "/usr/lib/python3.4/distutils/command/build_ext.py", line 339, in run self.build_extensions() File "/usr/lib/python3.4/distutils/command/build_ext.py", line 448, in build_extensions self.build_extension(ext) File "/usr/lib/python3.4/site-packages/setuptools/command/build_ext.py", line 183, in build_extension _build_ext.build_extension(self, ext) File "/usr/lib/python3.4/distutils/command/build_ext.py", line 500, in build_extension include_dirs=ext.include_dirs, File "/tmp/pip-build-gwez0e7l/matplotlib/setupext.py", line 785, in __get__ result = obj._hooks[self._name]() + result File "/tmp/pip-build-gwez0e7l/matplotlib/setupext.py", line 805, in include_dirs_hook import numpy File "/usr/lib/python3.4/site-packages/numpy/__init__.py", line 170, in <module> from . import add_newdocs File "/usr/lib/python3.4/site-packages/numpy/add_newdocs.py", line 13, in <module> from numpy.lib import add_newdoc File "/usr/lib/python3.4/site-packages/numpy/lib/__init__.py", line 17, in <module> from . import scimath as emath ImportError: cannot import name 'scimath' ----------------------------------------Command "/usr/bin/python3 -u -c "import setuptools, tokenize;__file__='/tmp/pip-build-gwez0e7l/matplotlib/setup.py';exec(compile(getattr(tokenize, 'open', open)(__file__).read().replace('\r\n', '\n'), __file__, 'exec'))" install --record /tmp/pip-bb_8_3xg-record/install-record.txt --single-version-externally-managed --compile" failed with error code 1 in /tmp/pip-build-gwez0e7l/matplotlib/$

There's apparently precompiled binaries for Python on Windows, but Python itself (3.5.2) can't be installed, failing with "0x80080005 server execution failed" and advising me to check its install log (which of course contains nothing whatsoever related). But it matters little, because there are not actually any precompiled windows packages anyway, either for 3.5 or the older 2.7 I still have installed for use with Golly.

So currently I'm downloading an 800 MiB bundle called Python(x, y) which bundles matplotlib and which I hope will finally actually work.

EDIT 2: matplotlib as bundled with Python(x, y) works, but Python(x, y) itself does not contain the python27.dll file needed for Python scripts in Golly. And unfortunately it also cannot coexist with a vanilla Python 2.7, so this isn't a solution either.

OTOH I was able to pip-install matplotlib on Cygwin's Python after installing the (unrelated, IMO, but what do I know) gtk 2.0 library/headers packages. After also installing statsmodels (which necessitated, among other things, LAPACK and a Fortran compiler) I whipped up this:

Which is a good start. Script:

#!/usr/bin/python3import numpy as npimport matplotlib.pyplot as pltimport statsmodels.api as sm# read data filedatafile=open("xs_scatterplot.dat","rb")x, y=np.loadtxt(datafile,delimiter="\t",skiprows=1,unpack=True)# generate a scatter plotfig, ax=plt.subplots()ax.scatter(x, y, label='Still lifes')ax.set_yscale('symlog', linthreshy=1)ax.set_xlabel('Population', fontsize=15)ax.set_ylabel("Count", fontsize=15)ax.set_title('Total still lifes', fontsize=20)plt.ylim(ymin=0)ax.grid(True)fig.tight_layout()# plot linear fitlogy=np.log(y)coefficients=np.polyfit(x, logy, 1)polynomial=np.poly1d(coefficients)ys=polynomial(x)plt.plot(x, np.exp(ys), label='Linear fit')# plot LOWESS fitlowess=sm.nonparametric.lowessz=lowess(logy, x, frac=0.3, it=3, return_sorted=False)plt.plot(x, np.exp(z), label=r'LOWESS fit, $\alpha=0.3$')# add legendplt.legend(loc='upper right')# save plot to filefig.set_size_inches(8, 8)plt.savefig("xs_scatterplot.png", dpi=80)

Data:

Population      Count4       384433800591905       53876157623626       235544426867417       62243883881908       11885732885689       633029573810      431592191611      127682180012      19256354809113      16280761814      7376654001615      43686175116      275936537417      14524693418      51915078019      1821341220      6491821021      150793322      958602323      46892924      42276725      21624926      14674927      3172828      302792329      926630      4533231      1768032      728433      18134      30435      2436      5537      938      41039      440      526541      242      6545      146      1

I'm fairly happy with how sane matplotlib's defaults are and how good its plots look without much tweaking -- most of the time required for this was installing Python modules, figuring out issues with the same, and trying to understand matplotlib's documentation.
If you speak, your speech must be better than your silence would have been. — Arabian proverb

Catagolue: Apple Bottom • Life Wiki: Apple Bottom • Twitter: @_AppleBottom_

Proud member of the Pattern Raiders!

Apple Bottom

Posts: 1023
Joined: July 27th, 2015, 2:06 pm

### Re: Scatter plot for still lifes (R help needed)

Hmm... So you use python from cygwin with golly? That's very unique. Vanilla python for windows installs matplotlib without a problem:
PS C:\Users\Scorbie> python -m pip install matplotlibCollecting matplotlib  Downloading matplotlib-1.5.1-cp35-none-win32.whl (6.2MB)    100% |################################| 6.2MB 126kB/sCollecting numpy>=1.6 (from matplotlib)  Downloading numpy-1.11.1-cp35-none-win32.whl (6.6MB)    100% |################################| 6.6MB 125kB/sCollecting pytz (from matplotlib)  Downloading pytz-2016.6.1-py2.py3-none-any.whl (481kB)    100% |################################| 481kB 1.4MB/sCollecting pyparsing!=2.0.4,>=1.5.6 (from matplotlib)  Downloading pyparsing-2.1.5-py2.py3-none-any.whl (42kB)    100% |################################| 51kB 3.7MB/sCollecting python-dateutil (from matplotlib)  Downloading python_dateutil-2.5.3-py2.py3-none-any.whl (201kB)    100% |################################| 204kB 2.4MB/sCollecting cycler (from matplotlib)  Downloading cycler-0.10.0-py2.py3-none-any.whlCollecting six>=1.5 (from python-dateutil->matplotlib)  Downloading six-1.10.0-py2.py3-none-any.whlInstalling collected packages: numpy, pytz, pyparsing, six, python-dateutil, cycler, matplotlibSuccessfully installed cycler-0.10.0 matplotlib-1.5.1 numpy-1.11.1 pyparsing-2.1.5 python-dateutil-2.5.3 pytz-2016.6.1 six-1.10.0

Edit That's a very decent short script indeed! couldn't get statsmodels library to work in windows, but except that it's kinda nice
Note for readers: the xs_scatterplot.dat file has its tabs converted to spaces by phpBB.
Best wishes to you, Scorbie

Scorbie

Posts: 1376
Joined: December 7th, 2013, 1:05 am

### Re: Scatter plot for still lifes (R help needed)

Scorbie wrote:Hmm... So you use python from cygwin with golly? That's very unique. Vanilla python for windows installs matplotlib without a problem:[/code]

No, Golly's using the regular Windows version of Python. But when working on the command line I prefer Cygwin, and use Cygwin's python there.

Scorbie wrote:Edit That's a very decent short script indeed! couldn't get statsmodels library to work in windows, but except that it's kinda nice
Note for readers: the xs_scatterplot.dat file has its tabs converted to spaces by phpBB.

Thanks! And good to know about phpBB playing tricks there. (I suppose I could use commas instead of tabs when generating that file anyway; the only reason I used tabs is because that's what R liked best.)
If you speak, your speech must be better than your silence would have been. — Arabian proverb

Catagolue: Apple Bottom • Life Wiki: Apple Bottom • Twitter: @_AppleBottom_

Proud member of the Pattern Raiders!

Apple Bottom

Posts: 1023
Joined: July 27th, 2015, 2:06 pm

### Re: Scatter plot for still lifes (R help needed)

BTW, just for fun I now generated images for not just B3/S23/C1 but B3/S23/!C1 (i.e. all other symmetries combined) as well. These are all using data from October 1st (right now; MediaWiki does not support linking to stable versions of the current revision of an uploaded file, so these will always point to the latest image):

I'm honestly not sure the LOWESS fit in particular is actually meaningful, but the plots sure are interesting to look at.

For the data and scripts used, see the file description pages on the wiki, linked e.g. from the Catagolue article.
If you speak, your speech must be better than your silence would have been. — Arabian proverb

Catagolue: Apple Bottom • Life Wiki: Apple Bottom • Twitter: @_AppleBottom_

Proud member of the Pattern Raiders!

Apple Bottom

Posts: 1023
Joined: July 27th, 2015, 2:06 pm

### Re: Scatter plot for still lifes (R help needed)

Wow, seriously nice work with the plots and the wiki article!
Best wishes to you, Scorbie

Scorbie

Posts: 1376
Joined: December 7th, 2013, 1:05 am

### Re: Scatter plot for still lifes (R help needed)

Scorbie wrote:Wow, seriously nice work with the plots and the wiki article!

Aww shucks, sugarcube. Thank you!
If you speak, your speech must be better than your silence would have been. — Arabian proverb

Catagolue: Apple Bottom • Life Wiki: Apple Bottom • Twitter: @_AppleBottom_

Proud member of the Pattern Raiders!

Apple Bottom

Posts: 1023
Joined: July 27th, 2015, 2:06 pm