## Scatter plot for still lifes (R help needed)

For discussion directly related to ConwayLife.com, such as requesting changes to how the forums or wiki function.
Apple Bottom
Posts: 1030
Joined: July 27th, 2015, 2:06 pm
Contact:

### Scatter plot for still lifes (R help needed)

I was thinking it would be nice to have a scatter plot of (total) still life counts vs. populations in the Catagolue article on the LifeWiki (which is why I'm sticking this in "Website Discussion", BTW), akin to the upper plot on page 2 of the full results for Andrzej Okrasinski's census.

That sort of thing's best done programmatically, so I turned to R, which I vaguely recalled being told was useful; but I've never used that before myself, and found it rather unwieldy, so I was wondering if anyone here either a) was an R wizard who could help improve the following plot, or b) had suggestions for a better tool.

For now here's the current best attempt at a plot I have:

Here's the R script used:

Code: Select all

library(bit64)
options(scipen=100)
attach(stilllifes)
## svg("xs_scatterplot.svg", width=10, height=10, antialias="subpixel")
png("xs_scatterplot.png", width=640, height=640, antialias="cleartype")
plot(Population, Count, log="y", lab=c(length(Population), ceiling(log10(max(Count)))+1, 7), xaxs="i")
## text(Population, Count, Count, cex=0.6, pos=4, col="red")
legend("topright", inset=0.05, title="Total still lifes", c("Count", "Linear fit", expression(paste("LOWESS fit, ", alpha, "=0.3"))), pch=c(21, -1, -1), lty=c(0, 1, 1), col=c("black", "black", "blue"))
title(main="Total still lifes by population")
abline(lm(log10(Count)~Population))
lines(lowess(stilllifes, f=0.3), col="blue")
dev.off()

And finally, the data file (note that this was generated from somewhat older data; I'd use newer data when generating an image to upload to the LifeWiki):

Code: Select all

Population	Count
4	37230663244700
5	5217667971195
6	22811439059639
7	6028058512999
8	1151083489759
9	6130605168
10	4179800708
11	1236556937
12	186490260778
13	157670526
14	71439981537
15	423074176
16	2672307142
17	140665171
18	502771638
19	17640474
20	62870208
21	1460833
22	9283930
23	454126
24	409529
25	209479
26	142231
27	30761
28	2932260
29	8988
30	43911
31	17135
32	7052
33	177
34	300
35	23
36	51
37	9
38	402
39	4
40	5092
41	2
42	64
45	1
46	1

So, yeah. Any tips on how to improve this plot? R seems to offer precious little control over formatting, axes, labels, ticks etc., but maybe I'm just looking in the wrong places.

Alternatively, any other (better) tools to generate such plots? Programs that can be used non-interactively (i.e. called with a script as part of a shell script) would be very strongly preferred.
If you speak, your speech must be better than your silence would have been. — Arabian proverb

Catagolue: Apple Bottom • Life Wiki: Apple Bottom • Twitter: @_AppleBottom_

Proud member of the Pattern Raiders!

Scorbie
Posts: 1445
Joined: December 7th, 2013, 1:05 am

### Re: Scatter plot for still lifes (R help needed)

Python + matplotlib?
Disclaimer: I have only heard of it.
Best wishes to you, Scorbie

Apple Bottom
Posts: 1030
Joined: July 27th, 2015, 2:06 pm
Contact:

### Re: Scatter plot for still lifes (R help needed)

Scorbie wrote:Python + matplotlib?
Disclaimer: I have only heard of it.
"matplotlib tries to make easy things easy and hard things possible" -- that certainly sounds promising. (Also, they're taking more than a subtle cue from Perl there, aren't they.)

I'll look into it, thanks!

EDIT: not much luck so far. Cygwin doesn't provide a package, and the pip package can't even be installed:

Code: Select all

$pip install matplotlib [...] running build_ext building 'matplotlib.ft2font' extension Traceback (most recent call last): File "<string>", line 1, in <module> File "/tmp/pip-build-gwez0e7l/matplotlib/setup.py", line 277, in <module> **extra_args File "/usr/lib/python3.4/distutils/core.py", line 148, in setup dist.run_commands() File "/usr/lib/python3.4/distutils/dist.py", line 955, in run_commands self.run_command(cmd) File "/usr/lib/python3.4/distutils/dist.py", line 974, in run_command cmd_obj.run() File "/usr/lib/python3.4/site-packages/setuptools/command/install.py", line 61, in run return orig.install.run(self) File "/usr/lib/python3.4/distutils/command/install.py", line 539, in run self.run_command('build') File "/usr/lib/python3.4/distutils/cmd.py", line 313, in run_command self.distribution.run_command(command) File "/usr/lib/python3.4/distutils/dist.py", line 974, in run_command cmd_obj.run() File "/usr/lib/python3.4/distutils/command/build.py", line 126, in run self.run_command(cmd_name) File "/usr/lib/python3.4/distutils/cmd.py", line 313, in run_command self.distribution.run_command(command) File "/usr/lib/python3.4/distutils/dist.py", line 974, in run_command cmd_obj.run() File "/usr/lib/python3.4/site-packages/setuptools/command/build_ext.py", line 50, in run _build_ext.run(self) File "/usr/lib/python3.4/distutils/command/build_ext.py", line 339, in run self.build_extensions() File "/usr/lib/python3.4/distutils/command/build_ext.py", line 448, in build_extensions self.build_extension(ext) File "/usr/lib/python3.4/site-packages/setuptools/command/build_ext.py", line 183, in build_extension _build_ext.build_extension(self, ext) File "/usr/lib/python3.4/distutils/command/build_ext.py", line 500, in build_extension include_dirs=ext.include_dirs, File "/tmp/pip-build-gwez0e7l/matplotlib/setupext.py", line 785, in __get__ result = obj._hooks[self._name]() + result File "/tmp/pip-build-gwez0e7l/matplotlib/setupext.py", line 805, in include_dirs_hook import numpy File "/usr/lib/python3.4/site-packages/numpy/__init__.py", line 170, in <module> from . import add_newdocs File "/usr/lib/python3.4/site-packages/numpy/add_newdocs.py", line 13, in <module> from numpy.lib import add_newdoc File "/usr/lib/python3.4/site-packages/numpy/lib/__init__.py", line 17, in <module> from . import scimath as emath ImportError: cannot import name 'scimath' ---------------------------------------- Command "/usr/bin/python3 -u -c "import setuptools, tokenize;__file__='/tmp/pip-build-gwez0e7l/matplotlib/setup.py';exec(compile(getattr(tokenize, 'open', open)(__file__).read().replace('\r\n', '\n'), __file__, 'exec'))" install --record /tmp/pip-bb_8_3xg-record/install-record.txt --single-version-externally-managed --compile" failed with error code 1 in /tmp/pip-build-gwez0e7l/matplotlib/$


There's apparently precompiled binaries for Python on Windows, but Python itself (3.5.2) can't be installed, failing with "0x80080005 server execution failed" and advising me to check its install log (which of course contains nothing whatsoever related). But it matters little, because there are not actually any precompiled windows packages anyway, either for 3.5 or the older 2.7 I still have installed for use with Golly.

So currently I'm downloading an 800 MiB bundle called Python(x, y) which bundles matplotlib and which I hope will finally actually work.

EDIT 2: matplotlib as bundled with Python(x, y) works, but Python(x, y) itself does not contain the python27.dll file needed for Python scripts in Golly. And unfortunately it also cannot coexist with a vanilla Python 2.7, so this isn't a solution either.

OTOH I was able to pip-install matplotlib on Cygwin's Python after installing the (unrelated, IMO, but what do I know) gtk 2.0 library/headers packages. After also installing statsmodels (which necessitated, among other things, LAPACK and a Fortran compiler) I whipped up this:

Which is a good start. Script:

Code: Select all

#!/usr/bin/python3

import numpy as np
import matplotlib.pyplot as plt
import statsmodels.api as sm

datafile=open("xs_scatterplot.dat","rb")

# generate a scatter plot
fig, ax=plt.subplots()
ax.scatter(x, y, label='Still lifes')
ax.set_yscale('symlog', linthreshy=1)
ax.set_xlabel('Population', fontsize=15)
ax.set_ylabel("Count", fontsize=15)
ax.set_title('Total still lifes', fontsize=20)
plt.ylim(ymin=0)
ax.grid(True)
fig.tight_layout()

# plot linear fit
logy=np.log(y)
coefficients=np.polyfit(x, logy, 1)
polynomial=np.poly1d(coefficients)
ys=polynomial(x)
plt.plot(x, np.exp(ys), label='Linear fit')

# plot LOWESS fit
lowess=sm.nonparametric.lowess
z=lowess(logy, x, frac=0.3, it=3, return_sorted=False)
plt.plot(x, np.exp(z), label=r'LOWESS fit, $\alpha=0.3$')

plt.legend(loc='upper right')

# save plot to file
fig.set_size_inches(8, 8)
plt.savefig("xs_scatterplot.png", dpi=80)

Data:

Code: Select all

Population      Count
4       38443380059190
5       5387615762362
6       23554442686741
7       6224388388190
8       1188573288568
9       6330295738
10      4315921916
11      1276821800
12      192563548091
13      162807618
14      73766540016
15      436861751
16      2759365374
17      145246934
18      519150780
19      18213412
20      64918210
21      1507933
22      9586023
23      468929
24      422767
25      216249
26      146749
27      31728
28      3027923
29      9266
30      45332
31      17680
32      7284
33      181
34      304
35      24
36      55
37      9
38      410
39      4
40      5265
41      2
42      65
45      1
46      1

I'm fairly happy with how sane matplotlib's defaults are and how good its plots look without much tweaking -- most of the time required for this was installing Python modules, figuring out issues with the same, and trying to understand matplotlib's documentation.
If you speak, your speech must be better than your silence would have been. — Arabian proverb

Catagolue: Apple Bottom • Life Wiki: Apple Bottom • Twitter: @_AppleBottom_

Proud member of the Pattern Raiders!

Scorbie
Posts: 1445
Joined: December 7th, 2013, 1:05 am

### Re: Scatter plot for still lifes (R help needed)

Hmm... So you use python from cygwin with golly? That's very unique. Vanilla python for windows installs matplotlib without a problem:

Code: Select all

PS C:\Users\Scorbie> python -m pip install matplotlib
Collecting matplotlib
100% |################################| 6.2MB 126kB/s
Collecting numpy>=1.6 (from matplotlib)
100% |################################| 6.6MB 125kB/s
Collecting pytz (from matplotlib)
100% |################################| 481kB 1.4MB/s
Collecting pyparsing!=2.0.4,>=1.5.6 (from matplotlib)
100% |################################| 51kB 3.7MB/s
Collecting python-dateutil (from matplotlib)
100% |################################| 204kB 2.4MB/s
Collecting cycler (from matplotlib)
Collecting six>=1.5 (from python-dateutil->matplotlib)
Installing collected packages: numpy, pytz, pyparsing, six, python-dateutil, cycler, matplotlib
Successfully installed cycler-0.10.0 matplotlib-1.5.1 numpy-1.11.1 pyparsing-2.1.5 python-dateutil-2.5.3 pytz-2016.6.1 six-1.10.0

Edit That's a very decent short script indeed! couldn't get statsmodels library to work in windows, but except that it's kinda nice
Note for readers: the xs_scatterplot.dat file has its tabs converted to spaces by phpBB.
Best wishes to you, Scorbie

Apple Bottom
Posts: 1030
Joined: July 27th, 2015, 2:06 pm
Contact:

### Re: Scatter plot for still lifes (R help needed)

Scorbie wrote:Hmm... So you use python from cygwin with golly? That's very unique. Vanilla python for windows installs matplotlib without a problem:[/code]

No, Golly's using the regular Windows version of Python. But when working on the command line I prefer Cygwin, and use Cygwin's python there.
Scorbie wrote:Edit That's a very decent short script indeed! couldn't get statsmodels library to work in windows, but except that it's kinda nice
Note for readers: the xs_scatterplot.dat file has its tabs converted to spaces by phpBB.
Thanks! And good to know about phpBB playing tricks there. (I suppose I could use commas instead of tabs when generating that file anyway; the only reason I used tabs is because that's what R liked best.)
If you speak, your speech must be better than your silence would have been. — Arabian proverb

Catagolue: Apple Bottom • Life Wiki: Apple Bottom • Twitter: @_AppleBottom_

Proud member of the Pattern Raiders!

Apple Bottom
Posts: 1030
Joined: July 27th, 2015, 2:06 pm
Contact:

### Re: Scatter plot for still lifes (R help needed)

BTW, just for fun I now generated images for not just B3/S23/C1 but B3/S23/!C1 (i.e. all other symmetries combined) as well. These are all using data from October 1st (right now; MediaWiki does not support linking to stable versions of the current revision of an uploaded file, so these will always point to the latest image):

I'm honestly not sure the LOWESS fit in particular is actually meaningful, but the plots sure are interesting to look at.

For the data and scripts used, see the file description pages on the wiki, linked e.g. from the Catagolue article.
If you speak, your speech must be better than your silence would have been. — Arabian proverb

Catagolue: Apple Bottom • Life Wiki: Apple Bottom • Twitter: @_AppleBottom_

Proud member of the Pattern Raiders!

Scorbie
Posts: 1445
Joined: December 7th, 2013, 1:05 am

### Re: Scatter plot for still lifes (R help needed)

Wow, seriously nice work with the plots and the wiki article!
Best wishes to you, Scorbie

Apple Bottom
Posts: 1030
Joined: July 27th, 2015, 2:06 pm
Contact:

### Re: Scatter plot for still lifes (R help needed)

Scorbie wrote:Wow, seriously nice work with the plots and the wiki article!
Aww shucks, sugarcube. Thank you!
If you speak, your speech must be better than your silence would have been. — Arabian proverb

Catagolue: Apple Bottom • Life Wiki: Apple Bottom • Twitter: @_AppleBottom_

Proud member of the Pattern Raiders!