Several people have asked in the telegram chat for the steps of the "gunbot console" I made so I can monitor multiple pairs on one screen at once. What this does is let me see the most important recent information of 16 pairs on one screen. I have dual-screens, so I can see 32 pairs on the two screens. All at once - for example, I can see if an instance needs to be restarted. Or I can see if polo is giving me errors across all of them at once (the dreaded 429 errors). Or I can see if one of the pairs is getting the "polo is being bad and giving us wrong alt coin balance!!!!!!!!" and I need to restart the bot.
All these things help me manage multiple pairs with less effort and with a keen eye.
This is what it looks like on one screen:
And for another one with just 6 pairs, it would look like this (for example, ETH only has a few available coin pairs): You might have another 16 pairs on the second screen, or yet another 16 pairs behind that, or whatever.
This is draft #1 - I was able to make this setup work for me in a few minutes, because I've used screen, tee and multitail in a former life and it's second nature. However, it's not necessarily easy to explain to others who are new to it.
I'm working on a draft #2 for an improved console, but it will take a lot longer. Since draft #1 works and suits its purpose and only took a few minutes, I might not hurry too much for draft #2
I've color coded the text in the console like so:
RED TEXT: Error codes like status 422, 500, 520, 420
Yellow Text: Bags. There's a "has order: " line in the output. That means there's a bag.
Purple Text: There's an active BUY order still at polo
Cyan Text: There's an active SELL order still at polo
Red/Yellow Text: Order information (buy or sell order placed), profit calculated.
Red/White text: The price is about to buy or about to sell. Might want to watch because something's happening.
Green Text: supergun true and falls no
RED TEXT: supergun false and falls yes
For example, if Gunbot is thinking about buying or thinking about selling, you'll see the ">= prSell" or "<= prBuy" turn red with white background so it will catch your attention.
The whole point is that I can watch the screen at 16 different pairs and only look for major things - bags, balance, purchases, market swings, buys, sells, etc. And then I can launch another 16 pairs and another 16 pairs, etc. The general setup is:
- `screen` is a program that lets you have multiple terminals and gunbot will run while you're logged out.
- Each gunbot instance runs in a different screen terminal
- Each gunbot instance pipes output to `tee`
- `tee` is a program that sends output to console AND puts output into a file
- `multitail` is a neat program that will tail a file and put it into a self-contained window and do things with it
- There are tons of command line options you need for multitail to make it pretty
- I wrote a multitail.conf that strips out the extra stuff we don't need
- For my purposes, I wrote a bash script for each group of up to 16 pairs of gunbot instances
Also, I have no idea if this will work for anyone else - it works for me, so that's all I know. So, some caveats:
- This is working for me. I make no warranty for anyone else.
- This is based on v2.0.3b - probably should work with other versions
- This is using stepgain output - probably should work with other versions
- I'm using Ubuntu LTS 14.04 - probably should work with other versions
- Not expected to work with Windows in any way
In my environment, I create a new directory for each gunbot instance, and I name the directory with a naming convention using the pair name with a dash. For example, "btc-maid" or "eth-gnt". This is important in order for the rest of this to work smoothly!
------------------------------
GNU SCREEN:
------------------------------
As I mentioned above, I use
GNU screen to manage multiple gunbot instances. Google how to install screen for your OS. It's pretty easy using the package manager of your choice. There are some good tutorials about screen
here,
here and
here.
I typically will group no more than 16 instances in a single screen session. For one, I have a pretty big monitor and fitting more than 16 together gets to be a little crowded. I probably could fit 20 and maybe 24 but I'm not running that many pairs yet. Maybe when I do, I'll try it. You're free to try. When things are working, each cycle only really needs 4-5 lines of output so you could easily increase the number of output in multitail or use a smaller font. It's doable.
Put this in a file called ~/.screenrc - modify at your leisure or find some other tidbits online. For example, vim ~/.screenrc or use nano or your favorite text editor.
This is the .screenrc I use:
defscrollback 9999
startup_message off
shell -$SHELL
hardstatus string '%{= kG}%{C}Screen: %{c}%{= kG}[%= %{= kw}%?%-Lw%?%{r}(%{W}%n*%f%t%?(%u)%?%{r})%{w}%?%+Lw%?%?%= %{g}][%{C} %d-%m %{W}%c %{g}]'
caption always "%{= kw}%-w%{= BW}%n %t%{-}%+w %-= %c"
vbell off
termcapinfo xterm* 'is=\E[r\E[m\E[2J\E[H\E[?7h\E[?1;4;6l'
Now you can start screen. Just type in
# screen
and you should have a blank window with a "0 bash" at the bottom
now you have one bash shell running and it's screen number is zero "0".
If you press CTRL+a and then the letter "c" for "create" it will create a new window and run a bash shell by default and it will be screen number "1" (aka "window 1")
You can keep doing this (CTRL+a, then letter c, or CTRL+a-c for short) and you will keep creating new screen windows. Notice how it keeps adding little tabs at the bottom, and each time it increases the number next to it. Each window is a new separate bash shell and you can run whatever you want in it (like, say, a gunbot instance).
CTRL+a is the "command key" for screen. It's how you tell screen what to do. CTRL+a-? (CTRL+a then question mark) will give you all the help, and there are lots of tutorials online that will help guide you through all the power tools of what screen can do (tip: google search for "GNU screen" instead of just "screen"). Here are the main ones you need for gunbot:
- CTRL+a-space - moves forward one window
- CTRL+a-backspace - moves backward one window
- CTRL+a-CTRL+D - disconnect from screen session altogether (so you can reconnect later)
- screen -r this resumes a disconnected screen (assuming you only have one disconnected session - see tutorial links above)
- CTRL+a-(number) - connect to window number specified (single digit only unfortunately)
------------------------------
USING TEE WITH GUNBOT
------------------------------Part of how this console magic works is that we need gunbot's output but we only want part of it. So, we pipe gunbot's output to a program called "tee". Go to one of the screen windows and run gunbot like you normally do but pipe the output to tee.
Now, what I do is I use a naming convention, not only for each directory I run gunbot in
(important) but also the name of the file the I pipe the console output to
(important). That way the multitail works more easily.
Also, I keep everything in a directory called 'gunbot' and each instance is in its own subdirectory labeled with the pair, say "btc-maid".
So, I go to window 1 (NOT WINDOW 0 because I use window 0 for my "status console") -
Press CTRL+a-1 (this will get you to window 1 no matter what window you're 1) and it will highlight the tab at the bottom with the "1 bash").
zcash@zcash1:~$ cd gunbot
zcash@zcash1:~/gunbot$ cd btc-maid
zcash@zcash1:~/gunbot/btc-maid$ ./gunbot BTC_MAID | tee -a btc_maid-console.log
This specific example I'm showing above is for gunbot v2.0.3. It launches stepgain, loads the configuration from BTC_MAID-config.js (which is just how it works in this version), displays the output into the console, and will also put all the output into the btc_maid-console.log file.
Then press CTRL+a-space to go to the next window (or CTRL+a-c to create one if I need a new window) and repeat the steps for the next pair - create a new btc-(pair) directory if needed... untar the whole gunbot installation files if needed....
QUICK TIP for spinning up a new pair: what I do is create a master installation directory when Gunthar gives us a stable version and I know I'm going to use it.. for example, v2.0.3b is what I'm using now. So I create a gunbot/v2.0.3 directory that is blank except for the files needed.
So, if I want a new pair, say btc-NEW, I do the following:
zcash@zcash1:~/gunbot$ mkdir btc-new
zcash@zcash1:~/gunbot$ cd v2.0.3
zcash@zcash1:~/gunbot/v2.0.3$ tar cvf - . | (cd ../btc-new; tar xfBp -)
./
./config.js
./package.json
./BTC_DASH-config.js
./gunbot
./save.json
./x86/
./x86/gunbot
zcash@zcash1:~/gunbot/v2.0.3$ cd ../btc-new
zcash@zcash1:~/gunbot/btc-new$ mv BTC_DASH-config.js BTC_NEW-config.js
zcash@zcash1:~/gunbot/btc-new$ vim !$
This creates a nice pristine copy that I can edit. Then I edit the config of the new coin (if needed) and I'm ready to add it somewhere and run it. This could be scripted if I was doing it often enough.
------------------------------
STATUS SO FAR
------------------------------So, hopefully you were able to continue with the above steps and launch your multiple gunbot instances, each in their own screen window. Keep doing CTRL+a-space to move forward a window at a time so you can watch gunbot happily run (don't worry that you don't see any colors from gunbot... tee doesn't support colors... and you don't want colors in the console logs anyway because it messes up the multitail coding).
Ok, now let's say we have 8, 12, 16 or so gunbot instances running in the single screen session. For many people, that's enough. If you didn't want "cryptowally's fancy linux console" and just wanted screen to have the ability to run gunbot in unattended fashion, you're done. You can just do the following:
- CTRL+a-space to move forward and see the raw console of each instance
- CTRL+a-backspace to move backwards
- CTRL+a-CTRL+D to detach your entire screen session - that way everything continues to run while you're logged out.
- screen -r to restore or return to your screen session later (more advanced options exist here, like naming them with -S and -list)
If you still want the fancy console, continue reading.
------------------------------
MULTITAIL CONSOLE
------------------------------
Again, I use a main "gunbot" directory that holds all my pairs. Depending on how you do your pairs, this may change for you. You might make a "btc1" to hold the first 16 pairs, and a "btc2" for the next 16 pairs and a "eth1" for the eth pairs, etc.
You need to have multitail installed. Start at the
official home page first, then google for some of the other tutorials. Here are some of my favorites.
Installation of multitail on linux tips, A good .conf file, and
Stackexchange article on watching a file even if not created. Create a "multitail.conf" file that contains the following text:
#include:/etc/multitail.conf
#check_mail:0
colorscheme:gunbot
cs_re:red,magenta:< prBuy
cs_re:red,white:<= prBuy
cs_re:red,white:>= prSell
cs_re:red,yellow:> prSell
cs_re:red,yellow:Sell order placed
cs_re:red,yellow:.*ORDER INFORMATION.*
cs_re:red,yellow:.*profit.*
cs_re:white:.*MinPriceToSell.*
cs_re:white:.*MaxPriceToBuy.*
cs_re:white:.*MaxPriceToSell.*
cs_re:white:.*MinPriceToBuy.*
cs_re:yellow:^Direction:\s+\d+
cs_re:white,red:Not enough BTC
cs_re:red:Error\!*
cs_re:red:.*failed.*
cs_re:red:.*no altcoins.*
cs_re:red:statusCode (.*)
cs_re:green:true
cs_re:red:false
cs_re:magenta:.*buy.*
cs_re:cyan:.*sell.*
cs_re:red:falls\syes
cs_re:green:grows\syes
cs_re:yellow:Has order:.*
cs_re:white:\:\:.*
Of course, the format is pretty self explanatory - if you want different colors, change the colors and the text that you want it to match. It's normal regex so if you understand regular expressions it's pretty easy. Put the matches in order from top to bottom.
Notice I'm also highlighting times when there isn't enough BTC or ETH, or when SELL or BUY orders are placed. Those are things I like to have pointed out to me if I'm just watching things go by.
The next thing I like to see are a list of my latest trades. So I wrote a little bash script that finds them for me. Since they're in the console logs, I can find them.
Create a directory called stats and create a file in there called "latesttrades.sh":
#!/bin/sh
for i in `ls /home/zcash/gunbot/[be]*/[be]*-console.log`
do
tail -30000 $i | egrep '(Buying|Selling|Error\!\!\! Not enough )'
done
Note that you will need to change the "/home/zcash/gunbot/" part to be your normal directory.... and for me, since I run btc and eth pairs, the be be part is a regular expression that matches both "btc" and "eth" in the name. What it does is looks for ALL the files with "console.log" in the name, but only the ones with "btc" or "eth" in the directory name, AND in the file name but only ones that start with "b" or "e" at the beginning of the name... and then it looks at the last 30000 lines of the file and finds if it bought, sold or had an error.
That's what gives me a nice one-liner summary at the bottom right hand corner of the console -- not sure if you noticed that in the first picture. If you don't want that, you can leave out the script.
----------------------
The big stats page
----------------------
Now for the script that fires off the multitail. Create a "btcstat.sh" file inside the top-level gunbot directory and put the following in it:
#!/bin/sh
multitail -Z red,black,inverse --config multitail.conf -CS gunbot -x "%m %u@%h %f (%t) [%l]" -b 4 -s 2 -m 0 -n 100 \
--mark-interval 200 \
-Ev "before to buy" \
-Ev "Cannot sell" \
-Ev "MaxPriceToBuy" \
-Ev "Last price <=" \
-Ev "Last price >=" \
-Ev "Last price >" \
-Ev "Last price >" \
-Ev "Altcoin Balance" \
-Ev "Not supergun:" \
-Ev "Supergun: " \
-Ev "avCoins" \
-Ev "CANCEL sell orders" \
-Ev "CANCEL buy orders" \
-Ev "SEll on start" \
-Ev "^$" \
-Ev "^\s+\d+\.\d+$" \
-Ev "No order" \
-Ev ":::" \
-Ev "Error\!\!\! statusCode 422" \
-Ev "Will repeat" \
-Ev "cycle" \
-Ev "No basePrice" \
-Ev "need_" \
-Ev "Response:" \
-Ev "Bot responsed" \
-Ev "next delay" \
-Ev "^\*" \
-Ev "\*\*" \
btc*/*-console.log \
-ts -c- -Rc 10 -l "./stats/latesttrades.sh "
In short, what this does is:
- runs multitail
- loads the multitail.conf file we created before
- loads the gunbot scheme
- will warn you with yellow marker lines if any window doesn't have output for 200 seconds (so you can see any HANGING gunbot instances so you can restart them manually... ie, CTRL+a-space over to that window, CTRL+C to stop gunbot, up-arrow and press enter...)
- strips out all the lines we don't need - that's what the -Ev lines do
- dynamically makes a window for any -console.log file that matches the btc*/ line (so make sure your directory structure is good)
- runs the stats/latesttrades.sh script every so often so you know how your trades are going
What *I* do is run ./btcstat.sh from screen window 0. So I launch all my gunbots starting on window 1, then 2, then 3... etc and make sure they're all running first... then go to window 0 and run ./btcstat.sh. That way all the -console.logs are there. multitail will create all the windows it needs.... if you have 8 console logs, it will create 4 x 2 windows plus the latesttrades, so it will have 9 windows. If you have 16 gunbots running, then you will have 17 windows (8 on the left and 9 on the right).
Notice again that if you start getting yellow lines in any of the windows, that means the gunbot instance in that specific window hasn't had any output for 200 seconds (or whatever you have "mark_interval" set to)...
See the window with the yellow in it? Multitail is telling me that gunbot has locked up or is hung. gunbot isn't responding or at least the console log file it's writing to hasn't had any updates.
I NEED TO ACT!So what do you do? You need to CTRL+a-space over to that window, CTRL-C gunbot, up arrow and press enter so it can restart. This happens sometimes when gunbot sent a request to polo and polo didn't send it back. Here's an example - you'll see "collecting data..." at the bottom of the window and gunbot is not doing anything... there's no output to the console like the other windows.
Just restart gunbot in that one window and you're back in business. (ie, CTRL++-space until you get there, then CTRL+C to stop gunbot. Press up-arrow to bring up the command that last started gunbot with the exact same settings in that window and press enter. It will restart with the same settings and same tee command and everything. Then CTRL+a-0 to get back to your btcstat.sh page.
I use that mark-interval and the yellow coloring so I can more quickly see that something is wrong. There have been times when I see a whole bunch of yellow. Thank goodness I had this system. I can much more quickly determine if I have a bot that's hung up because I can quickly scan all the bots and see what's wrong.
Now, if you're running any of the other strategies like BB, Russian Roulette, Supergun or some version other than 2.0.3b, I don't know how the console output is different from stepgain. I don't run those. There may be tweaks needed to multitail.conf file, btcstat.sh, or latesttrades.sh scripts. If someone wants to post their console output here, I can maybe post some possible tweaks.
-------------------
DISK SPACE CLEANUP
-------------------As you can imagine, having each gunbot instance write all its console logs to the disk can eat up a lot of disk space over time. You really only need the most recent console logs, so what you do with the logs after some time is up to you. But since you don't want your disks to fill up, you need something to clear out those files.
I use a tool called logrotate. If you want to copy what I do, it's easy and at least preserves the logs long-term by compressing them and only gives you a minor weekly task.
My system already had logrotate installed, so if you don't have it installed, google "logrotate" or "logrotated" and at least have the defaults running. Usually, /etc/logrotate.conf points to an /etc/logrotate.d directory where you can put your own local settings.
Create an /etc/logrotate.d/gunbot file and put the following, changing the directory to match your own console.log files and directory naming:
/home/zcash/gunbot/[be]*/*-console.log {
missingok
weekly
create 664 zcash zcash
rotate 30
compress
dateext
}
For example, if your home directory is /home/gunbot and you put all your files into a directory called btc1, and your pairs are btc-<coin>, then that first line would have /home/gunbot/btc1/btc-*/btc*-console.log in it. Just make sure the regular expression matches what you have there. You can use one line to match all your console logs if you want.
This will compress your big console.logs every week and rotate them out every month and put a date on them so if you wanted to go research how gunbot worked, you could still do so.
After my system has been running a while, here's how the log files look:
zcash@zcash1:~/gunbot$ ll btc-fct/
total 43952
drwxrwxr-x 3 zcash zcash 4096 May 9 09:23 ./
drwxrwxr-x 33 zcash zcash 4096 May 10 10:56 ../
-rw-rw-r-- 1 zcash zcash 3432 May 9 09:23 BTC_FCT-config.js
-rw-rw-r-- 1 zcash zcash 2586115 May 10 12:29 btc_fct-console.log
-rw-rw-r-- 1 zcash zcash 885421 Apr 16 06:54 btc_fct-console.log-20170416.gz
-rw-rw-r-- 1 zcash zcash 594402 Apr 23 21:49 btc_fct-console.log-20170424.gz
-rw-rw-r-- 1 zcash zcash 430755 Apr 30 06:40 btc_fct-console.log-20170430.gz
-rw-rw-r-- 1 zcash zcash 398708 May 7 06:53 btc_fct-console.log-20170507.gz
Notice how I still have my main console.log (and it's pretty big), and every week, it has a .gz file of the previous week's logs, which is much smaller.
I should point out that I enabled debug logging in my config... so I even have more console logging enabled than most people. You can tweak your logrotate file to delete the console logs after a week, or not keep so many if you'd like. Check out the many logrotate example files on the internet, or in the man pages.
Note that what this means is that multitail will turn yellow every week, because it won't know to refresh and look at the new log files. So Monday morning when you come in, you'll see a sea of yellow windows. Do not panic - just go to window 0, CTRL+C the btcstat.sh and up-arrow restart it so it can re-read the new console log files. Or, you may need to just go to each gunbot instance and restart them - just to refresh the tee overwriting the console logs. Don't worry - gunbot works fine and hasn't quit. It's just that logrotate and tee or multitail don't get along sometimes. It's still trading just fine.
--------------------
Multiple screen sessions
--------------------So, the above is for one screen session for up to 16 pairs. If you're going past that, I'd recommend you start the next set of pairs with another completely different directory and duplicate everything separately so nothing stomps on anything else. That way, you're free to use the asterisks and regular expressions and things without messing things up. It gets complicated if you're trying to juggle too many pairs in the same directory structure.
Personally, I run separate screens for my BTC and ETH pairs. When I start my BTC screen session the first time, I name the screen session "btc" by running it with "screen -S btc". For eth pairs, I do "screen -S eth". This is only the v
ery first time I launch screen, before I've started any gunbot sessions. That way, with the screen session itself having a name, when I do a "screen -list" I can reconnect to the one I want:
zcash@zcash1:~/gunbot$ screen -list
There are screens on:
31870.eth (04/05/2017 08:14:10 PM) (Attached)
31307.btc (04/05/2017 08:04:40 PM) (Attached)
Instead of using "eth" or "btc," you might use "btc1" or "btc2" for the first set of btc pairs, second set or whatever.
So, when I reconnect to my server, I can just type in "screen -r eth" and I'll see all my 6 eth screens still running. Then I fire up another putty session to my server and in that putty screen I type in "screen -r btc" and the 16 gunbot sessions show up there. Everything's still running and money's still being made.
Nothing's stopping you from running multiple screens for each set of pairs. Have a separate directory for each set of pairs. "btc1" would contain multitail.conf, btcstat.sh, and one subdirectory for each pair. Everything would be contained together so nothing stomped on each other.
btc2 would be a different directory with it's own multitail.conf, btcstat.sh and subdirectory for its pairs.
-----------------------
Next steps / future ideas
-----------------------Future Idea #1: better consoleI've been working on a Part II, where the output is more formatted instead of just filtered, and one where it can react to output (such as restarting gunbot when it's yellow), or maybe even change the configs if it notices things like BTC rising, but that takes time. Plus Gunthar keeps improving the bot (and changing the output) so at some point it might not be worth making changes.
But here's what I've mocked up as the next version of the console - if you'd like me to work on this, consider leaving a tip, or reputation. The next version constantly watches each cycle's output and feeds a realtime console with the updates but only gives the pertinent and relevant information. Do I have a bag? Is it about to buy? Do I have enough balance? Is the coin pumping? Are my settings way off? How come I don't understand bitcoin market? And my favorite: Why am I expecting Gunthar to read my mind when I haven't even tried to understand what gunbot is really doing?
I think with this kind of console, we can better tweak our settings and understand what's going on. Notice how in the console, I see consistent information in the same spot for each pair, the price is in the same format, I can see buy/sell history, and current orders, and when something changes, it highlights it with color, and I can see the last error, etc. All this is in the current console output, so I just need to scrape it, collect it, and then fill out the formatter.
Future Idea #2 - auto-launch. One thing I'm mostly done with is automating the above, where instead of manually setting up the screen sessions and such, one could use a .screenrc that launches the different gunbots in a synchronized manner, cd'ing to the directories and running gunbot correctly, and in a staggered fashion, say 3 seconds apart, so as not to anger the API gods. I just haven't finished it because it's not bothering me too much. I start gunbot by hand and time it manually by doing CTRL+a-space, up-arrow, enter and go to the next one and that takes about 3 seconds, and I do it so infrequently that I haven't found the need to automate it. But it can be done and I know others have done it.
But it would look something like this:
$ more testscreen/fireguns
startup_message off
defscrollback 100000
#create windows
screen -t btc_gnt
select 0
stuff "sleep 3; fire gunbot command here"
screen -t btc_nav
select 1
stuff "sleep 3; fire gunbot command here"
# repeat to add more and more gunbot windows
# .......
screen -t stats
select 16
stuff "multitail monitoring command here"
#
Note you need a CTRL+M sequence at the end of the stuff commands in order for it to actually execute the commands... like so:
--------------
TIP JAR
--------------If you enjoyed this tutorial and would like to leave a small tip, you may do so at the following address:
(I'll take any altcoin tip as well.. PM me to tell me the coin and I'll give you an address)
BTC: 1AhkpVhEqxfiaJoEayifwgqppEqZVbdCmn
Show Posts
