Michael Leuchtenburg
United States
Cambridge
Massachusetts
flag msg tools
badge
Avatar
mbmbmbmbmb
At last, the first public release of GenetiTrade has arrived. With it, there's a wiki, a bug/feature request tracking system, and so forth.

You can check it out here: http://trac.slashhome.org/genetitrade/

If you know how to use GenetiTrade, I'd appreciate some help writing user docs. Just drop by the documentation page and start editing: http://trac.slashhome.org/genetitrade/wiki/Documentation

If you're interested in contributing code to GenetiTrade, please, do! Just submit a patch as an attachment to a ticket in Trac and we'll take a look at it.
 
 Thumb up
1.00
 tip
 Hide
  • [+] Dice rolls
Jeff Michaud
United States
Longwood
Florida
flag msg tools
On-Line Want List Generator - Hopefully Making Math Trades a Little Bit Easier
badge
Captain Kirk, Captain Picard, Captain Sisko, Captain Janeway, Captain Archer
Avatar
mbmbmbmbmb
dyfrgi wrote:
If you're interested in contributing code to GenetiTrade, please, do! Just submit a patch as an attachment to a ticket in Trac and we'll take a look at it.
Should patches be as a context diff (ie. "diff -c") so the UNIX patch command can be used?

I was wondering if a CVS repository would make sense?

Sounds good!

Jeff
 
 Thumb up
 tip
 Hide
  • [+] Dice rolls
David Spitzley
United States
Belleville
Michigan
flag msg tools
Avatar
mbmbmbmbmb
Any suggestion of how to open .tar.gz files in Windows XP?
 
 Thumb up
 tip
 Hide
  • [+] Dice rolls
Jeff Michaud
United States
Longwood
Florida
flag msg tools
On-Line Want List Generator - Hopefully Making Math Trades a Little Bit Easier
badge
Captain Kirk, Captain Picard, Captain Sisko, Captain Janeway, Captain Archer
Avatar
mbmbmbmbmb
Dspitzle wrote:
Any suggestion of how to open .tar.gz files in Windows XP?
www.winzip.com

or if you have www.Cygwin.com installed:

tar --gunzip xf file.tar.gz
 
 Thumb up
 tip
 Hide
  • [+] Dice rolls
Michael Leuchtenburg
United States
Cambridge
Massachusetts
flag msg tools
badge
Avatar
mbmbmbmbmb
A unified diff is my personal preference. That's diff -u, or svn diff (it uses the unified format by default).

There's a subversion repository set up there. http://svn.slashhome.org/genetitrade/ is the project root. It's currently mentioned under "downloads".
 
 Thumb up
 tip
 Hide
  • [+] Dice rolls
Michael Leuchtenburg
United States
Cambridge
Massachusetts
flag msg tools
badge
Avatar
mbmbmbmbmb
Dspitzle wrote:
Any suggestion of how to open .tar.gz files in Windows XP?

I suggest 7-Zip, if you want something with a GUI: http://www.7-zip.org/

Otherwis, as Jeff said, use tar from Cygwin.
 
 Thumb up
 tip
 Hide
  • [+] Dice rolls
Jeff Michaud
United States
Longwood
Florida
flag msg tools
On-Line Want List Generator - Hopefully Making Math Trades a Little Bit Easier
badge
Captain Kirk, Captain Picard, Captain Sisko, Captain Janeway, Captain Archer
Avatar
mbmbmbmbmb
dyfrgi wrote:
A unified diff is my personal preference. That's diff -u, or svn diff (it uses the unified format by default).

There's a subversion repository set up there. http://svn.slashhome.org/genetitrade/ is the project root. It's currently mentioned under "downloads".
Wow, am I behind the times! I had never heard of unified diff before, nor SVN being the next generation CVS!

In any case, svn is part of the Cygwin environment so hopefully I'm all set. Anyone else want info on SVN see http://subversion.tigris.org/
 
 Thumb up
 tip
 Hide
  • [+] Dice rolls
Jeff Michaud
United States
Longwood
Florida
flag msg tools
On-Line Want List Generator - Hopefully Making Math Trades a Little Bit Easier
badge
Captain Kirk, Captain Picard, Captain Sisko, Captain Janeway, Captain Archer
Avatar
mbmbmbmbmb
dyfrgi wrote:
There's a subversion repository set up there. http://svn.slashhome.org/genetitrade/ is the project root. It's currently mentioned under "downloads".
Using the command given on that page, ie:

svn checkout http://svn.slashhome.org/genetitrade/trunk

I'm being asked for a password. So I must have to register somehow... but I only see a "login" link on the site. Do you have to register folks on your end?
 
 Thumb up
 tip
 Hide
  • [+] Dice rolls
Michael Leuchtenburg
United States
Cambridge
Massachusetts
flag msg tools
badge
Avatar
mbmbmbmbmb
Whoops, sorry about that. It's fixed now - it should let you do a checkout without authenticating.
1 
 Thumb up
 tip
 Hide
  • [+] Dice rolls
Michael Van Biesbrouck
Canada
St Catharines
Ontario
flag msg tools
designer
badge
Avatar
mbmbmbmbmb
After experimentation in the last math trade I agree that a -e value that might take hours to evaluate is worthwhile, yet the improvements are in the endgame. To solve the patch problem I used -c 5 -e 5. Defaults and -e 4 just weren't as effective. I believe that the best solution was using a nonstandard -e argument. I don't think that the startup cost would sit well with most play-at-home users, but their computation might be largely irrelevant without longer edge extensions.
 
 Thumb up
 tip
 Hide
  • [+] Dice rolls
David Spitzley
United States
Belleville
Michigan
flag msg tools
Avatar
mbmbmbmbmb
When will a Windows executable be available? I've got some test data I'd like to let it chew on, but I don't have Python on hand.
 
 Thumb up
 tip
 Hide
  • [+] Dice rolls
Michael Leuchtenburg
United States
Cambridge
Massachusetts
flag msg tools
badge
Avatar
mbmbmbmbmb
mlvanbie wrote:
After experimentation in the last math trade I agree that a -e value that might take hours to evaluate is worthwhile, yet the improvements are in the endgame. To solve the patch problem I used -c 5 -e 5. Defaults and -e 4 just weren't as effective. I believe that the best solution was using a nonstandard -e argument. I don't think that the startup cost would sit well with most play-at-home users, but their computation might be largely irrelevant without longer edge extensions.

Figuring out how to compute that table faster, or how to do without it, is the primary focus of our current algorithmic development. One of my current goals is to figure out a method to measure the effectiveness of operators.

Dspitzle wrote:
When will a Windows executable be available? I've got some test data I'd like to let it chew on, but I don't have Python on hand.

When I find the time to figure out how to make one. Or someone else figures out how to make one. Probably some time in April.

Installing Python is really easy. It'd probably take you about 5 minutes. Just download this and run it: http://www.python.org/ftp/python/2.5/python-2.5.msi
 
 Thumb up
 tip
 Hide
  • [+] Dice rolls
Jeff Michaud
United States
Longwood
Florida
flag msg tools
On-Line Want List Generator - Hopefully Making Math Trades a Little Bit Easier
badge
Captain Kirk, Captain Picard, Captain Sisko, Captain Janeway, Captain Archer
Avatar
mbmbmbmbmb
Remote config files, persistent cache of cycles/....
By the way, should we continue to use the original thread:

http://www.boardgamegeek.com/thread/154926

for discussion of features and development? Or should we move discussions over to here?

Regarding computing tables and "play at home" math trades... I'm thinking that one of the things that a "distributed" math trade would do would be able to load pre-computed (by the person who is organizing the trade) cycles and edge extensions, a long with the want lists, so that the runs from play at home participants wouldn't duplicate work.

As far as distributed/play-at-home, I was thinking the play at home folks would start Genetitrade something like:

python genetitrade.py http://somewhere.net/xxx.xml (or .txt)

xml would make more sense long term, but unless a tool is provided to easily generate it, a simple plain text configuration file would be easier to setup for the average organizer.

The configuration file would have the url's for files containing the wants, and pre-computed cycles and edge extensions. Other info would be things like a an optional polling interval to see if the wants file has changed (which would mean the pre-computed cycles/edge... should have as well) so the clients (play at home runs of genetitrade) would also detect that they need to start from scratch. An optional deadline for play at home results (so the clients can stop automatically).

By the way, in general, even before adding support for easy distirbuted/play-at-home features, I think a nice feature would be to cache to disk the pre-computed cycles and edge extensions and use them if they are still valid if genetitrade is rerun with the same input (wants). For example, if the input file is named ../foo/bar.txt when save the computed cycles and edge extentions to bar.cache.txt (ie. strip any directory prefix and use the basename so cache is in current wd). Also in the cache would be a checksum/hash of the (preferably canonical) wants, and either implicitly or explicitly what the max cycle length is that's cached and the max length of the edge extensions.

On startup if the cache already exists, and the checksum of the wants matches the current intput file, load it. If the current run specifies using shorter lengths for the cycles and/or edge extensions, ignore the excess. If the current run specifies usign longer than what's cached, load what is there, then only compute the extra chain lengths and/or edge extensions needed, then rewrite the cache.

This cache file of course is what could then be copied to a web server and pointed at by a config file as mentioned above for distributed/play-at-home use.
 
 Thumb up
 tip
 Hide
  • [+] Dice rolls
James Perry
United States
Spring Hill
Tennessee
flag msg tools
badge
Ex-Term-In-Ate
Avatar
mbmbmbmbmb
JeffyJeff wrote:
Regarding computing tables and "play at home" math trades... I'm thinking that one of the things that a "distributed" math trade would do would be able to load pre-computed (by the person who is organizing the trade) cycles and edge extensions, a long with the want lists, so that the runs from play at home participants wouldn't duplicate work.

Unless I am missing something, isn't part of the "random" nature the cycle generation?

JeffyJeff wrote:
As far as distributed/play-at-home, I was thinking the play at home folks would start Genetitrade something like:

python genetitrade.py http://somewhere.net/xxx.xml (or .txt)

xml would make more sense long term, but unless a tool is provided to easily generate it, a simple plain text configuration file would be easier to setup for the average organizer.
Agreed.

JeffyJeff wrote:
The configuration file would have the url's for files containing the wants, and pre-computed cycles and edge extensions. Other info would be things like a an optional polling interval to see if the wants file has changed (which would mean the pre-computed cycles/edge... should have as well) so the clients (play at home runs of genetitrade) would also detect that they need to start from scratch. An optional deadline for play at home results (so the clients can stop automatically).
Why not compile the wants & pre-computed items into the config file? That would be far easier on the moderator as they wouldn't have to first upload two files before building the config file to get the correct URLs.

JeffyJeff wrote:
By the way, in general, even before adding support for easy distirbuted/play-at-home features, I think a nice feature would be to cache to disk the pre-computed cycles and edge extensions and use them if they are still valid if genetitrade is rerun with the same input (wants). For example, if the input file is named ../foo/bar.txt when save the computed cycles and edge extentions to bar.cache.txt (ie. strip any directory prefix and use the basename so cache is in current wd). Also in the cache would be a checksum/hash of the (preferably canonical) wants, and either implicitly or explicitly what the max cycle length is that's cached and the max length of the edge extensions.

On startup if the cache already exists, and the checksum of the wants matches the current intput file, load it. If the current run specifies using shorter lengths for the cycles and/or edge extensions, ignore the excess. If the current run specifies usign longer than what's cached, load what is there, then only compute the extra chain lengths and/or edge extensions needed, then rewrite the cache.

This cache file of course is what could then be copied to a web server and pointed at by a config file as mentioned above for distributed/play-at-home use.
Interesting idea.
 
 Thumb up
 tip
 Hide
  • [+] Dice rolls
Michael Leuchtenburg
United States
Cambridge
Massachusetts
flag msg tools
badge
Avatar
mbmbmbmbmb
The cycle caching is non-random; in 0.1, the edge caching uses some randomness. That could easily be changed if desired, though. The only reason for the randomness is to experiment with longer edge lengths with incomplete tables.

However, caching the edge extensions on a website may be difficult. You might notice that, when using a high edge extension count, GenetiTrade uses a lot of memory. On the order of 500MB, when running with length 5 edge extensions. Most of that RAM goes into storing the edge extension table.

Presumably this could be brought down a lot with a different storage scheme, compression, etc., but assuming that it can be brought down to 50MB, that's still quite large to store on a central server. 5MB would probably be reasonable.

I think that the only way we'll be able to do distributed computation of edge extensions is to build a P2P network with the nodes in it. That shouldn't be too difficult - we'll just use an existing library for the purpose. The P2P network will also be used for exchanging genomes, if that turns out to be helpful.

Regardless of whether distributed computation of cached cycles and edge extensions turns out to be feasible/easy to implement, fetching a configuration file/wantlist from a website and polling for updates is a feature I'd like to have in GT. I wonder if it'd be possible to keep most of the cached edge extensions when pulling minor updates? That'd be an interesting task.
 
 Thumb up
 tip
 Hide
  • [+] Dice rolls
davean scies
msg tools
Regai wrote:

JeffyJeff wrote:
As far as distributed/play-at-home, I was thinking the play at home folks would start Genetitrade something like:

python genetitrade.py http://somewhere.net/xxx.xml (or .txt)

xml would make more sense long term, but unless a tool is provided to easily generate it, a simple plain text configuration file would be easier to setup for the average organizer.
Agreed.

Not by me, I don't think XML would be a very good solution at all; how many config files have you ever seen that are XML? Not many, and probably none meant to be edited by hand.

XML is bulky and difficult to deal with, I think we should just us a standard config file which has the added benefit of not having to wrap parser to deal with and jsut getting values back from it.
 
 Thumb up
 tip
 Hide
  • [+] Dice rolls
James Perry
United States
Spring Hill
Tennessee
flag msg tools
badge
Ex-Term-In-Ate
Avatar
mbmbmbmbmb
davean wrote:
Regai wrote:

JeffyJeff wrote:
As far as distributed/play-at-home, I was thinking the play at home folks would start Genetitrade something like:

python genetitrade.py http://somewhere.net/xxx.xml (or .txt)

xml would make more sense long term, but unless a tool is provided to easily generate it, a simple plain text configuration file would be easier to setup for the average organizer.
Agreed.

Not by me, I don't think XML would be a very good solution at all; how many config files have you ever seen that are XML? Not many, and probably none meant to be edited by hand.

XML is bulky and difficult to deal with, I think we should just us a standard config file which has the added benefit of not having to wrap parser to deal with and jsut getting values back from it.

I've seen plenty, but I agree that they were not meant to be edited by hand. Thus Jeff's comments about there would have to be a config tool.

For the purposes of GenetiTrade, a simple text config file is probably best.

[edit]forgot to add:

However, the cached edges and/or cycles would 100% be best stored in an XML document if the inteded purpose is sending them across the web.
 
 Thumb up
 tip
 Hide
  • [+] Dice rolls
davean scies
msg tools
Regai wrote:

However, the cached edges and/or cycles would 100% be best stored in an XML document if the intended purpose is sending them across the web.

If you are planning on sending them over a network and actually want them to get there, you are definitely talking a binary file. XML would at a minimum (with compression) double the size of an already several hundred MB file.

To get this transmitted you'd definitely be looking at a domain specific compression algorithm and if there is any XML it is just encapsulating a binary file for no real purpose.

Remember the data we are talking about here is hundreds of millions of 2 to 7 integer units.

It probably doesn't make sense to transmit this information at all though. Best to keep it locally and lower the communication between nodes.
 
 Thumb up
 tip
 Hide
  • [+] Dice rolls
Stephen Waits
United States
San Diego
California
flag msg tools
mbmbmbmbmb
The way to distribute this is to basically let all participants run locally. Consider each instance a 'species' or 'deme'. As participants join, or depart, the server determines neighbors; probably on toroidal basis (i.e. wraps around).

After each generation, or each X generations, clients send their strongest individuals to the server, and receive the best individuals neighboring demes produced from the server. ("X" can be controlled dynamically by the server, based on load). These 'migrants' are then incorporated into the local population.

It's a relatively simple scheme, but should be very effective at distributing this problem.

Of course, some amount of authentication (signatures) should be included.

As well as GT is doing now, I think it'd do at least slightly better with something like this.

--Steve
 
 Thumb up
 tip
 Hide
  • [+] Dice rolls
Michael Leuchtenburg
United States
Cambridge
Massachusetts
flag msg tools
badge
Avatar
mbmbmbmbmb
swaits wrote:
The way to distribute this is to basically let all participants run locally. Consider each instance a 'species' or 'deme'. As participants join, or depart, the server determines neighbors; probably on toroidal basis (i.e. wraps around).

A sort of "island chain" model. It reminds me of those species, arrayed around a geographical feature (a lake or mountain range, say) wherein each can breed with the adjacent members, but if you take two diametrically opposite specimens, they can't breed. I wonder what sort of speciation could be observed in a genetic search system?

This would certainly slow down the propagation of good solutions, thus providing even more time for different high quality solutions to evolve without being trounced by an earlier-emerging competitor. Since GenetiTrade's distributed computation, when introduced, will probably use a server, it should be easy to experiment with different topologies for the islands. Well, as easy as anything taking a day or so to get results is to explore.
 
 Thumb up
 tip
 Hide
  • [+] Dice rolls
Stephen Waits
United States
San Diego
California
flag msg tools
mbmbmbmbmb
Yes, that's the idea. It's speciation on a grander scale. I think the migration should actually be kept fairly low in this case, as you also suspect.

I'd help you do it, but whenever I code Python, a small part of me dies.

I'm still very interested in coding your algorithm in C++. That I could do very quickly as long as I can carve out a chunk of time in which to do it; however, I still haven't quite decoded it all. (Again, Python).

--Steve
 
 Thumb up
 tip
 Hide
  • [+] Dice rolls
Michael Leuchtenburg
United States
Cambridge
Massachusetts
flag msg tools
badge
Avatar
mbmbmbmbmb
New release of GenetiTrade posted: version 0.1.2.

http://trac.slashhome.org/genetitrade/wiki/Downloads

CHANGE SUMMARY:
* New output header format (thanks JeffyJeff)
* Windows executable and installer
 
 Thumb up
 tip
 Hide
  • [+] Dice rolls