Sciencemadness Discussion Board - Anyone have a Linux or OS X machine w/32 GB RAM?

Anyone have a Linux or OS X machine w/32 GB RAM?

Polverone - 8-10-2015 at 16:33

And willing to install software and run some calculations on it?

I have been working with a former member here to develop composite thermochemical models based on the open source NWChem software:

https://github.com/mattbernst/composite-thermochemistry-nwch...

This is software to predict the enthalpy of formation of small molecules from (almost) first principles. The enthalpy of formation is e.g. <a href="http://link.springer.com/article/10.1007%2Fs10573-007-0074-6">directly useful as a predictor for propellant/explosive performance</a>. Calculated enthalpies for a compound and its decomposition/combustion products can also be used as <a href="https://www.sciencemadness.org/whisper/viewthread.php?tid=11195">inputs to more sophisticated models</a> for energetic material performance.

We are currently stuck because calculations on some of the open shell systems in the calibration test set require more RAM than either of us have available. We both have 16 GB on our largest machine, and a few calculations need more than that. I think that 32 GB will be sufficient to run all examples from the G2 reference set of compounds, though maybe not if we add the G3 set.

There are some details of the models that are unclear from the original papers, and we hope that completing the full test set will allow us to understand those details or at least refit the empirical correction terms for maximum accuracy. I'd also happily credit you (if desired) in the README for the software.

We'd be looking for someone who can install NWChem from source and our models from github and then run the calculations that we can't. I can provide directions for either Linux or OS X. Extremely adventurous Windows users with a ton of RAM might be able to participate via virtualized Linux.

annaandherdad - 8-10-2015 at 17:24

I would help if I could but I'm limited to 8GB. I imagine the codes are complicated. I wonder what "almost" first principles means.

aga - 9-10-2015 at 04:01

I have a tower PC at home that's unused.

Will see how much RAM it has and if it can be upgraded.

Would 64Gb RAM be more than enough including the G3 set ?

How much hard drive space is needed ?

Polverone - 9-10-2015 at 08:57

By "almost" first principles I mean that most of the model parameters are system-independent and come from basic definitions of atoms, molecules, and quantum mechanics*. There are two important empirical inputs: a table of experimental spin-orbit corrections for atomic species, and the so-called Higher Level Correction term. The HLC term is intended to correct for the slow convergence of the exact electronic energy toward the limit as basis set size is increased. It consists of an empirical adjustment to the system energy using one term for each valence electron pair and a different term for each unpaired valence electron. The actual terms are derived by an error-minimizing fitting to a set of experimental energies across a test set of systems.

Initially we thought that we could just reuse the already-published HLC terms. As long as every other part of our procedure is specified the same, the best HLC fitting should yield the same terms also.

Some months later, we're less confident in that. Curtiss and coworkers originally used the proprietary software Gaussian to implement their various composite thermo schemes. Today Gaussian 09 still has the most widely used implementation of the G1-G4 schemes. But Gaussian has some odd and poorly documented quirks. I've already been able to determine that Gaussian 09 deviates slightly from the procedures as originally published, for reasons unknown. A source code license to Gaussian costs a fortune so I can't see how the inner parts of the black box actually work. We're trying to run the full published G2 test set and extract every detail so we can pinpoint the differences between our current implementation and the original publications, and between the original publications and the current Gaussian implementation that everyone uses.

A 64 GB machine might be able to handle the full G3 test set. I don't recall at the moment what the most demanding system in that set is. If an eager participant had a 64 GB machine to contribute I'd certainly try to run some tests with my collaborator!

*Or rather, the computationally tractable approximations to quantum mechanics that are implemented in quantum chemistry software.

aga - 9-10-2015 at 10:40

Nope. The old tower i got looks too old and slow to get the job done.

Would this be any good ?

http://www.ebay.co.uk/itm/Barebones-Dell-Precision-490-Tower...

Polverone - 9-10-2015 at 11:03

Yes, that machine looks OK. I've received a private response from a member with access to an academic compute cluster, so don't spend your money on eBay just yet. Of course this other member hasn't yet managed to build NWChem from source. You've reminded me that I should consider second-hand hardware if I want to have a machine of my own for this purpose.

careysub - 9-10-2015 at 11:05

I have an 8 core 4 GHZ Linux computer with 32 GB of RAM.

Polverone - 9-10-2015 at 11:47

Great! Are you willing to participate? If so, make sure that you can build the NWChem 6.6 beta including Python support:

http://www.nwchem-sw.org/index.php/Download
http://www.nwchem-sw.org/index.php/Compiling_NWChem

I have personal experience building NWChem under Debian, Ubuntu, and OS X. Trouble on other Linux distributions will probably need to be referred to the NWChem users' forum: http://www.nwchem-sw.org/index.php/Special:AWCforum

careysub - 9-10-2015 at 11:55

Quote: Originally posted by Polverone

Great! Are you willing to participate? If so, make sure that you can build the NWChem 6.6 beta including Python support:

http://www.nwchem-sw.org/index.php/Download
http://www.nwchem-sw.org/index.php/Compiling_NWChem

I have personal experience building NWChem under Debian, Ubuntu, and OS X. Trouble on other Linux distributions will probably need to be referred to the NWChem users' forum: http://www.nwchem-sw.org/index.php/Special:AWCforum

I can give it a try. Currently I think I have XUBUNTU on it, but this an Ubuntu distribution.

Polverone - 9-10-2015 at 12:09

Here's how I build NWChem on my Ubuntu machine.

I have <a href="https://github.com/xianyi/OpenBLAS/archive/v0.2.14.tar.gz">OpenBLAS 0.2.14</a>, single-threaded version with 64 bit integer interface, installed under /opt/science/openblas/64/.

Here's how I built OpenBLAS:

Code:

make USE_THREAD=0 INTERFACE64=1
make install PREFIX=/opt/science/openblas/64

I build NWChem under /opt/science/nwchem/ like so:

Code:

cd nwchem-6.6
export USE_NOFSCHECK=TRUE
export TCGRSH=/usr/bin/ssh
export NWCHEM_TOP=`pwd`
export NWCHEM_TARGET=LINUX64
export NWCHEM_MODULES="all python"
export LARGE_FILES=TRUE
export ENABLE_COMPONENT=yes
export PYTHONHOME=/usr
export PYTHONVERSION=2.7
export PYTHONLIBTYPE=so
export PYTHONCONFIGDIR=config-x86_64-linux-gnu
export CC=gcc
export FC=gfortran
export FOPTIMIZE="-O3 -march=native -mtune=native -msse -msse2 -msse3 -mssse3 -msse4 -msse4.1 -msse4.2 -funroll-loops -fprefetch-loop-arrays -fvariable-expansion-in-unroller -ffast-math -mfpmath=sse"
export COPTIMIZE="-O3 -march=native -mtune=native -msse -msse2 -msse3 -mssse3 -msse4 -msse4.1 -msse4.2 -funroll-loops -ffast-math -mfpmath=sse"
export HAS_BLAS=yes
export BLASOPT="-L/opt/science/openblas/64/lib -lopenblas"
export USE_MPI=y
export USE_MPIF=y
export USE_MPIF4=y
export ARMCI_NETWORK=SOCKETS
export MPI_LOC=/usr/lib/openmpi/
export MPI_LIB=/usr/lib/openmpi/lib
export MPI_INCLUDE=/usr/lib/openmpi/include
export LIBMPI="-lpthread -L$MPI_LIB -lmpi_f90 -lmpi_f77 -lmpi"
export MRCC_METHODS=y
export CCSDTQ=y
export CCSDTLR=y
export IPCCSD=y
export EACCSD=y
cd $NWCHEM_TOP/src
make clean
make nwchem_config 2>&1 | tee make.nwchem_config.log
make &> build.log
cd ../contrib
./getmem.nwchem

This is a real "bells and whistles" build with a lot of optional capabilities. I think that you could omit the MRCC_METHODS, CCSDTQ, CCSDTLR, IPCCSD,
and EACCSD exports and the final binary would still do everything needed for the composite thermochemistry project.

aga - 9-10-2015 at 13:07

Just bought one of these :-

http://www.ebay.co.uk/itm/121727857740?_trksid=p2057872.m274...

Hopefully it'll do the OpenWRT stuff i need really fast, so the rest of the time it'll be idle.

When it arrives it can be mapped to a static with root access as a backup plan if required.

Polverone - 9-10-2015 at 13:32

<i>How much hard drive space is needed ?</i>

Ah, I didn't answer this before. You need maybe 10-15 GB free space to install NWChem and run calculations for every system in the current test set. The most demanding jobs are much more memory-intensive than disk-intensive, because NWChem was designed for high performance systems with a lot of RAM and there is little support for switching to disk-based storage.

In fact there is a bug in NWChem that forces the use of so much memory for these open-shell calculations. It is supposed to be possible to use a somewhat slower but much more memory-frugal scheme for the critical step that I can't fit in 16 GB, but that scheme is currently broken. A developer has pledged to fix it but it's not a high priority. The primary "customers" for NWChem have an incredible abundance of RAM; a single node on PNNL's primary system has 128 GB, and most jobs run with memory pooled across many nodes.

[Edited on 10-9-2015 by Polverone]

aga - 9-10-2015 at 13:40

Python, so High Level Language programmers.

Tends to be the case these days, so RAM usage/Program Size is hidden from them.

My personal best was a polar co-ordinate plotting routine in 253 bytes of RAM on a 6502 CPU (assembler obviously).

The ~2Mhz (not GHz) CPU speed made the on-screen shape rotation a bit jerky, but pretty good for the time.

Polverone - 9-10-2015 at 13:43

No, the Python code's memory/CPU footprint barely registers. It's just used to provide high level calculation control and data extraction. It's the actual quantum chemistry calculations that are demanding, and those routines are all implemented in Fortran.

aga - 9-10-2015 at 13:44

Ah. Not so bloated then.

aga - 20-10-2015 at 10:53

The 32Gb RAM PC tower and a new TFT screen arrived today.

I'm lacking a DVI cable to hook up the video, which i will get tomorrow.

Ubuntu Server 14.04.3 ok ?

Dunno if this would be of help to smared as well, or if users can Share nicely etc etc.

[Edited on 20-10-2015 by aga]

Polverone - 20-10-2015 at 13:00

Yes, Ubuntu 14.04 should be fine. That's what I compiled NWChem under most recently.

aga - 20-10-2015 at 13:12

OK. I'll go and get the video cable tomorrow (plus a working keyboard and mouse !)

If the install goes smoothly then i'll map a public IP to it and you can have Telnet/SSH access with root privs as well sometime tomorrow.

U2U a username/password for that please.

It's located at the edge of a huge wifi network, so do not expect more than about 10Mb/s download speeds for the software/data you need to get.

I'll be just doing OpenWRT compiles on it, which it should eat in a few minutes, and those do not happen every month, let alone every day.

Power cuts do happen during big storms, and it's that season at the moment.

ahill - 20-10-2015 at 14:26

..sounds really interesting.

Is AWS an option ? If the other offers fall thru, I'd be happy to spin up a ~30Gig AWS server for a day or two.

8 - 16 CPUs with 30Gig RAM is 50 - 85c an hour - so if the whole thing only runs for a few hours - pretty cheap - if its a few months - not so much.

aga - 20-10-2015 at 14:28

Who Pays ? Would be the obvious question.

Are you offering this for free ?

hinz - 20-10-2015 at 14:31

If any more help is needed, I have a few SunFire X4440s here that are currently unused. (Quad 6-core AMD 8400 CPUs with 128GB PC-5300 DDR2 RAM each, HDD is between 200-500GB per computer)

I would have to install Xubuntu however, so it could take some time until fully operational

aga - 20-10-2015 at 14:43

Wow !

That would sound like the better option.

Do It now.

Install would just take an hour, max.

hinz - 20-10-2015 at 15:02

Here are the SunFire beasts. Absolutely sexy, if you ask me

Build from cheap parts whenever they popped up on ebay.
Downloading and burning Xubuntu at the moment.

Just have to tidy up, reroute some LAN cables and bring some stuff outside into storage.

Polverone - 20-10-2015 at 15:13

128 GB of RAM you say? I would be very interested if you could give me a ssh login for one of those beasts. I can provide a public key if you want to go passwordless.

bwn - 20-10-2015 at 15:24

Quote: Originally posted by hinz

Here are the SunFire beasts. Absolutely sexy, if you ask me

Build from cheap parts whenever they popped up on ebay.
Downloading and burning Xubuntu at the moment.

Just have to tidy up, reroute some LAN cables and bring some stuff outside into storage.

Slick looking machines, I've always liked Sun's style as well.

(Polverone - I was going to offer to scrounge up some extra ram for my server but it sounds like you've got a better offer!)

[Edited on 20-10-2015 by bwn]

hinz - 20-10-2015 at 15:38

Yes, 128GB with 24 cores that is

Just give me some time (a few hours and maybe help me with setup of remote login) to tidy up. I was busy fixing old test equipment in other part of the room (didn't need fancy computers for that), so that part of the room was used as temporary storage and is thereby quite messy.

Just tested my network 20Mbit down /4.5Mbit upload

hinz - 20-10-2015 at 17:28

Installing Xubuntu 14.04
...
Reinstalling, because wrong RAID array was selected as boot drive. :mad:

...
Xubuntu running fine, until I changed ATI driver from open source one to propietary in hope of getting second monitor active. Now X11 crashes after bootup. Have to change it back to open source one.

Since it's somewhat late now, I go sleeping, will be back in a few hours

[Edited on 21-10-2015 by hinz]

bwn - 20-10-2015 at 22:18

Quote: Originally posted by hinz

Installing Xubuntu 14.04
...
Reinstalling, because wrong RAID array was selected as boot drive. :mad:

Regarding the ATI driver: I'm using Debian on my machine with an ATI card, but I had similar sounding problems setting up the proprietary drivers. In my case, it seems to be the configuration file more often than not. This is my bookmark for Debian setup, but Ubuntu probably has a similar page:

https://wiki.debian.org/ATIProprietary#Configuration

ahill - 20-10-2015 at 23:10

@aga sure - that was what I was offering. Like I said - hours or days - not weeks or months.

The SunFires if otherwise unused do seem like a better option tho.

Polverone - 22-10-2015 at 08:40

Any further progress with the SunFire? I am excited at the prospect of trying it for these calculations.

aga - 22-10-2015 at 13:36

Mine arrived broken.

Ordered some spares to figure out what is busted, so no fallback plan at the moment.

hinz - 23-10-2015 at 13:59

OK, finally got one Sunfire running stable. Second one also running, while third one has trouble booting.

Had some trouble with graphics, since the X4440 doesn't seem to recognize (lspci doesn't show anything) my other two ATI V3900 graphic cards in PCIe x8 slots. (Would have been nice to have 3 monitors, now only have one monitor driven by one V3900 in x16 slot). Since that isn't a real problem with remote operation, I just go with one monitor for now.

Have downloaded NwChem 6.6 and extracted it to Desktop. Current private subnet setup is Fritzbox:192.168.178.1=>some Netgear Switch=>Sunfire X4440: 192.168.178.35

ssh and ssh-client is installed and Sunfire is SSH-able from subnet.
Now I need your help with compiling NWChem (you probably have some compiling options you want, among the -j24 flag

)
And you have to help me setting up remote SSH for you ( I probably have to set up port forwarding among other stuff in my Fritzbox)
Tell me what you need and I PM my public IP, logins and PWs.

Working Sunfire

[Edited on 23-10-2015 by hinz]

[Edited on 23-10-2015 by hinz]

aga - 23-10-2015 at 14:17

Nice !

Not sure what it is, but Nice one Hinz !

[Edited on 23-10-2015 by aga]

Polverone - 23-10-2015 at 15:31

Yes, you will need port forwarding set up. Put a long password on the account if you're going to run on the standard port 22; automatic password attacks are pretty aggressive.

You should be able to use the compilation instructions I posted: https://www.sciencemadness.org/whisper/viewthread.php?tid=63...

You will need python-dev installed and you should already have built OpenBLAS as shown in that post. If you run into trouble you can post here or set me up with a login so I can diagnose things interactively. It's been a while since I had to do initial machine setup for building NWChem. It's not complicated but I'm not sure I remember what the non-default dependencies are.

hinz - 23-10-2015 at 16:21

Compiled & installed OpenBLAS & NwChem in /opt/science/

Also set up port forwarding to SSH port 22 in my Fritzbox. But since I'm inside my subnet, I can't tell whether my X4440 is SSH-able from outside, so I've PM'd you my current IP, and login info. Tell me if it works for you.

BTW, NwChem folder in /opt/science/nwchem../bin is empty, so maybe something didn't work right.

Regards

[Edited on 24-10-2015 by hinz]

Polverone - 27-10-2015 at 19:03

A few updates: even with 128 GB of RAM, there are a number of calculations that won't run until the NWChem developers fix certain bugs that prevent economizing on memory. I was happily able to run a number of calculations that overwhelm my own machines.

I discovered on hinz's machine that CCSD(T) calculations can fail when there are too many cores allocated to small calculations. I have been able to add a heuristic to avoid most of these problems and a recovery strategy to automatically fix the rest after the failure is detected. I just pushed those changes to the public github repository. Thank you, hinz, for helping me improve this code.