Script and template .vhdx to install a fully functional apgsearch virtual machine (Also a Docker container too!)

[etmoonshade]

This is a project I was working on for a week or two, and I thought it might be useful to others. When I mentioned it in the apgsearch thread, it was suggested I post it here.

Edit from 2023:
Look, a Docker image!
https://hub.docker.com/r/etmoonshade/apgmerabase

I suspect it won't work if you just try to start it up with no config, but you can run it with the following command line:
docker run -d -e SOUPS=<soups> -e PAYOSHA_KEY=<key> -e THREADS=<#> etmoonshade/apgmerabase

SOUPS is the number of soups to submit per haul.
PAYOSHA_KEY is your Payosha256 key.
THREADS is the number of threads you want to run this on.


The documentation for the script and the VM is left below, and it should still be available.

---------------------------------

Summary/Requirements
This is a script which, when executed, will install a virtual machine running apgsearch, configured for B3/S23. The VM will be created with 2GB of RAM, with other parameters specified on the command line. apgsearch will automatically start when the script finishes configuring it.

System Requirements:
Multi-core processor
8+ GB of system RAM
~2.5GB free HDD space per VM (in theory, this could go up to 4GB per VM, but I've been running mine for about a week and haven't seen them grow much.)
7-Zip (downloadable from https://www.7-zip.org)
Windows 10 Pro, Enterprise, or Education. Windows Server 2016+ will also work. Windows Server 2012 may work, but is untested.
Hyper-V role must be installed.

Note: Hyper-V can mess with your network adapters in quaint and curious ways. If you're not comfortable fixing it if it breaks, please don't install it.

Please let me know if you run into any issues with using this, and I'll see if I can help out. One very specific thing I'm not sure of is how this will run on an Intel processor, since I built it out on AMD. It should be abstracted out by Hyper-V, but I've never dealt with migrating a VM between Intel and AMD before.

I've written the install instructions with the assumption that the user has basic skills with the command line. I'll try and shore up places where I could go into more detail if necessary.


Download Contents
The download includes three files:
* voidlinuxbase2.vhdx
This is the virtual machine itself. Uncompressed, it should be roughly 2.5GB. Unfortunately, I wasn't really able to strip down the base further. I'm currently working on an Arch linux install which would hopefully be much smaller, but this is what we've got for now.

* Install-APGSearchVM.ps1
This is the powershell script that does most of the magic. Usage instructions later, but as with ANY script you pick up on the internet, you should try to at least vaguely understand what it does before you run it.

* Get-APGStats.ps1
This is a little script that goes out to the various VMs installed and gets some stats on what they're actually doing.


Notes on the VM
Locations:
/usr/src/apgmera: This is where the apgluxe setup script lives.
/var/log/apgmera/*.log: This is where the output from apgluxe goes, and is the source used by the Get-APGStats script.
/usr/local/arpscan.sh: I'll avoid the rant about this for now, but this is a "run once" script that does an ARP scan on the entire subnet the VM is connected to. Harmless, but it could tweak the nose of your security folks if you've got them. See notes for an explanation, but suffice to say it's both dumb and necessary as far as I can tell.
/etc/sv/apgmera: This is where the "service file" lives. Probably not terribly relevant to most.

Relevant commands:
Void Linux is a bit special, command-wise. You'll be able to transfer some of your linux skills if you have 'em, but not everything.
sv up apgmera: Starts apgmera.
sv down apgmera: Stops apgmera.
touch /etc/sv/apgmera/down: Existence of this file forces the service to stay down after a reboot.


Using the install script
This is where the magic happens. Note that the script used is attached, if people want to take a look at it before downloading the whole mess.

Step 1: Unzip apgsearch-base.7z wherever you want to.

Step 2: Open powershell.exe as an administrator (or an account with the ability to create directories and virtual machines. It's probably administrator for most people.)

Step 2.5: Just realized this because I had to rebuild the server I made this on. Run the following command if you haven't done so on the server:

Code: Select all

Install-Module Posh-SSH

and follow the instructions to (probably) install NuGet and install from the "untrusted" repository (assuming you have no issues with that.)

No, I am not renumbering all of my steps. >:(

Step 3: Change directory to wherever you unzipped apgsearch-base.7z

Note the name of this .vhdx file - you'll need to provide this to the script.

Built-in help on using the script can be gotten with the following command:

Code: Select all

Get-Help .\Install-APGSearchVM.ps1 -full

Step 4: Run the command with whatever arguments you deem necessary.

General syntax is:

Code: Select all

Install-APGSearchVM.ps1 [[-userPassword] <String>] [[-logging] <Boolean>] [[-hostNumber] <Int32>]
[[-key] <String>] [-cores] <Int32> [-soups] <Int32> [-sourceDir] <String> [-destDir] <String> [<CommonParameters>]

A common use case would be something like the following:

Code: Select all

./Install-APGSearchVM.ps1 -userPassword SuperStrongPassw0rd -key youcanputinacustompayoshakeyright -cores 6 -soups 100000000 -sourceDir "C:\Downloads\apgsearch-base\voidlinuxbase2.vhdx" -destDir "C:\Virtual Machines\"

Six cores assigned to the VM (which means 5 cores used,) 100M soups per haul submitted, uses a Payosha256 key, assigns a specific password.

More examples (and many more details) are available with the built-in help.

Note: I don't recommend using logging unless you're comfortable maintaining it yourself. It just dumps it in the same directory as the apgluxe binary, and this will require more disk space - possibly more than the size of the disk I gave it. Basic logs (i.e. console output) are auto-rotated daily and deleted after 3 days.


Using the monitoring script
The monitoring script is more of an ad-hoc "what are you doing?" script that looks for any VM on the host with a name starting with "apgsearch".

The script takes two parameters: userName and userPassword.

If you just run ./Get-APGStats.ps1, it will prompt you for your username (root) and your password.

If you're going to be watching it frequently, an insecure way of making things easier is to set a couple of variables:

Code: Select all

$name = "root"
$userPass = "12345"

and then run the command in Powershell as follows:

Code: Select all

./Get-APGStats.ps1 -userName $name -userPassword $userPass

It'll pass through the values to the script.

Script output looks something like this:

Code: Select all

Connecting to apgsearch5
Connecting to apgsearch1
Connecting to apgsearch6
No useful logs retrieved - retrying apgsearch6
Connecting to apgsearch3
No useful logs retrieved - retrying apgsearch3
No useful logs retrieved - retrying apgsearch3
Connecting to apgsearch7
Connecting to apgsearch8
Connecting to apgsearch2
Connecting to apgsearch4
apgsearch5:     Average soups/sec: 24587.794    Current soups completed: 182600000
apgsearch1:     Average soups/sec: 25077.990    Current soups completed: 182900000
apgsearch6:     Average soups/sec: 25057.518    Current soups completed: 9900000
apgsearch3:     Average soups/sec: 24693.998    Current soups completed: 104400000
apgsearch7:     Average soups/sec: 25203.318    Current soups completed: 78600000
apgsearch8:     Average soups/sec: 25411.940    Current soups completed: 52500000
apgsearch2:     Average soups/sec: 24930.503    Current soups completed: 87000000
apgsearch4:     Average soups/sec: 24277.746    Current soups completed: 141100000

Overall Average soups/sec: 24905.100875
Total soups/sec: 199240.807

The "no useful logs" lines mean that either you've found about 40 infinite growth patterns in a row, or it's in the middle of validating someone else's haul. Those are just to let you know the script is still doing something. If for some reason you get more than about 20 of those lines in a row, you can use Ctrl+C to cancel the script. Whatever machine it was stuck on may have something wrong with it. Delete/rebuild is probably your best way forward if that happens.


So wait, where's the download?
Right here:
https://epsilonfilestorage.blob.core.wi ... ch-base.7z

Changed this to an Azure blob storage link. Let me know if you can't download it - there shouldn't be any issues with daily limits this way, but I might have mucked up the permissions.

Notes (or: Anticipatory FAQs)
The ARP scan
So, the ARP scan. Many popular/common Linux installations have what are called integration services - in this case, specifically integration with Hyper-V. This allows a crafty admin to get information about the virtual machine, like the IP address. Void Linux does not have integration services, which means I can't get the IP address of the virtual machine to do the last bits of setup required (e.g. changing password, modifying the apgmera launch script, etc.)

If you're installing this on an enterprise/educational/otherwise managed network, you'll want to talk to your system administrators first, just in case this sets off alarms for them (and really, you should be involving them anyway if you're installing this on a machine that isn't owned by you )


On a more technical note: After a couple of days of research (and being accused in at least one IRC channel of making a virus,) blasting the entire network seems to be the only solution that will actually get the IP address into the ARP table of the VM host (which I don't "know" the IP of.) Once it's in the ARP table, I can use the MAC address (which I can get from the network adapter, since the VM host is aware of it) to find the IP and continue the setup.

As I note in the script, this is incredibly dumb, but I don't have anywhere near enough skill to write my own integration services for an arbitrary Linux distribution.

Passwords
The Get-APGStats script expects all the passwords to be the same. If you don't care about logging into the machines after they're started, you don't need to specify a password when installing the VM - it'll auto-generate a pretty strong password for you and spit it out to the console. If you ever want to get in again (e.g. to update apgmera,) you'll want to record the password(s) somewhere though.

Why only use n-1 cores?
This was a specific design choice I made while testing the script. Assigning n cores to a VM but only using n-1 cores means that it's a lot less likely that the guest OR host operating system will get starved of CPU resources. If you've ever tried to fix a machine with a pegged CPU, you'll know exactly why I did this.

Someone familiar with Powershell or similar languages could look at line 284 in the script if they really want to modify this behavior.

Todo
Auto-update apgmera on the first install.
Add more examples to the script as I see "real life" use cases.

[Null]

Great job, and this would be the right thread when someone works out a docker image (or Dockerfile) for apgsearch :)

[etmoonshade]

I'd love to get apgsearch containerized, but I have zero clue how one would go about it. From what little I've read, it looks like someone would have to hack apgsearch itself to make it "compatible" with a container. I admit I don't have a good understanding of containers in the first place though.

Also, I know Alpine Linux is popular for containerization... but apgsearch runs terribly on Alpine. So a word to the wise if someone else decides to tackle that.

(specifically, Alpine uses musl libraries instead of glibc, and musl does NOT like how apgsearch does memory calls for whatever reason. This is one of the rabbit holes I went down when building this out, and is why I ended up going for a chunkier Void Linux install. I still need to do some tests, but performance may improve when using the LD_PRELOADER trick to use jemalloc instead of whatever is stock.)

[Null]

debian-slim or redhat's ubi8 image would be another option for glibc-based containers in case others are interested.

From what little I've read, it looks like someone would have to hack apgsearch itself to make it "compatible" with a container. I admit I don't have a good understanding of containers in the first place though. :)

I don't think it's necessary, but I'll try and report this if I have any time left.

[etmoonshade]

debian-slim or redhat's ubi8 image would be another option for glibc-based containers in case others are interested.

From what little I've read, it looks like someone would have to hack apgsearch itself to make it "compatible" with a container. I admit I don't have a good understanding of containers in the first place though.

I don't think it's necessary, but I'll try and report this if I have any time left.

it's only, y'know, 2 years later

With that said, hey, look, a Docker image:
https://hub.docker.com/r/etmoonshade/apgmerabase