Generator-Gunbot - Setup GUNBOT v3.3.2 in less than 5 minutes (linux)

[beer-k0in]

Hello everybody.

Working with version 3.3.2

First of all: Thanks a lot for all the tutorials in this forum, they helped me a lot! And of course thanks to Gunthar for this nice bot!

To have a better understanding of the GUNBOT and to have an easy start using it, I created
1. an installer to only use one command to setup GUNBOT 3.3.2 on a fresh virtual server and
2. a generator to init and start all the trade pair configs I want to use.

From creating a new server at DigitalOcean to let the bot start trading it takes less than 5 min. 

But:
I'm new to GUNBOT. So my approach has some downsides - for now

Installer:
1. The installer only works on Debian / Ubuntu (Choose Ubuntu 16.04.)
2. You need to be root user
3. I recommend a fresh virtual server

Generator:
1. Only supports trade pairs BTC to XXX
2. You can only change a few settings of the trade pair config
3. The other settings are predefined and very likely not perfect - but a good start


How does it work

Installer

The installer is basically a bash script. That script will

- update the system (apt upgrade),
- install node 6.x (for the generator and pm2),
- install required tools (yo, pm2, unzip),
- install the GUNBOT 3.3.2,
- creates some handy aliases,
- install the generator

You can view the whole script here: https://github.com/BeerK0in/generator-gunbot/blob/master/install.sh

Generator

The generator is using a tool called yeoman to create the trade pair config files and optionally starts the GUNBOT for the pairs.
It will ask you some questions you need to answer. Like:

- What market do you want to setup (poloniex, bittrex or kraken)
- Your Market API key
- Your Market API secret
- Max amount of BTC used by each pair per trade
- What BUY strategy do you want to use
- What SELL strategy do you want to use
- Questions about the selected strategies
- Your Delays of the Gunbot
- Select the currencies you want to trade
- Select the trade pair currencies you want to automatically start

And than it creates the config files and starts GUNBOT using pm2.

Aliases

The installer also sets up some aliases to do more with less.

- gcd: Go to the folder GUNBOT is installed
- gadd: Use the generator to add a new pair (*)
- gl: Get a list of all running GUNBOTs
- glog [botname or id]: Shows the log of the selected bot (botname = BTC_XXX, use gl to get id)
- gstop [botname or id]: Stops a running bot (botname = BTC_XXX, use gl to get id)
- gstart [botname or id]: Starts a stopped bot (botname = BTC_XXX, use gl to get id)

(*) For now gadd will use the same settings as defined in the first run of the generator. A next version will allow you to set specific strategies for new trade pairs.


Let's do it

1. New VPS. (optional)

I use DigitalOcean a lot, so I can give a brief tutorial:
- Login
- Create Droplet
- Select Ubuntu
- Select the 512MB RAM option for ~3 running trade pairs, 1GB for ~8 pairs or 2GB for ~17 pairs
- Choose San Francisco 2 as location
- Give it a name like 'my-gunbot'
- Click create

Here is an image of the whole creation page: https://i.imgur.com/4uxqjBJ.png
Next you get an email with the IP address of the new server and a password. Looks like this: https://i.imgur.com/LTxIos9.png
Use SSH to connect to this new server:

Code: [Select]

ssh root@123.123.123.123
- Accept the authenticity of the host (enter "yes")
- Set a new root password
- Done

Looks like this: https://i.imgur.com/lpsPoYS.png

If you want to create a VPS at DigitalOcean, you can use this affiliate link to get $10 credit at DigitalOcean: https://m.do.co/c/fade3d3435ba
Of course you can use any server you like.


2. Installer

After you connected via ssh to your new server (must be running Debian or Ubuntu) just run this command:

Code: [Select]

curl -qsL https://raw.githubusercontent.com/BeerK0in/generator-gunbot/master/install.sh | bash -- && exec bash
Two minutes later the installer is done.

3. Generator

Now run this command:

Code: [Select]

ginit
Enter your Poloniex API key and the secret, answer the questions and you are done.
Hint for 3.3.2: on a machine with 1GB start 8 or less bots.

You can run 'gl' to see all GUNBOTs.

=> That's it. 

Here a screen how it looks like. Depending on your selected strategy and trade pairs, it will look a little bit different.




To access the log of a bot simply enter

Code: [Select]

glog BTC_LTC
Or the id of that bot. In my example I could also just run 'glog 2'.



Maybe this helps some people to get into the GUNBOT.


Additional scripts

"gstop all" and "gstart all" will likely cause 422 or 429 errors on Kraken and Poloniex, so user cw was so kind and wrote two scripts that nicely and safely start and stop all the bots.
You should use them. Please read his post in this thread to add them to your system: https://gunthy.org/index.php?topic=277.msg2110#msg2110
Thanks cw!


Known Errors

ERROR! - There was an error starting GUNBOT for BTC_EXP:
1

There are 2 possible issues.

The easy one

Type 'gl' and check if the pair you want to start is already running. Would look like this:


If it is in that list and its state is online, type 'gstop BTC_XXX' to stop it before you run 'ginit'.

The nasty one

Sometimes the PM2 God Deamon got detached from your (root-)user and you will see an empty list if you enter 'gl' BUT if you run 'pgrep -f BTC_XXX' you will see a id a result.

So the Gunbot is running but pm2 does not want to see it.

You need to kill the PM2 God Deamon, and remove the content in ~/.pm2/:

Code: [Select]

pgrep -f PM2 | xargs kill -9
Wait like 2 to 5 min till all Gunbots also stopped.
Check if there are running Gunbots

Code: [Select]

pgrep -f gunthyIf the dont stop, also kill them

Code: [Select]

pgrep -f gunthy | xargs kill -9
Now remove the content of ~/.pm2/

Code: [Select]

rm -r ~/.pm2/*
And run run 'ginit' again.



BK

[msn100001]

Very cool...

[gg]

Select the 512MG Ram option for ~5 running bots, or 1GB for ~10 running bots

running bot means 1 currency pair?

[beer-k0in]

Select the 512MG Ram option for ~5 running bots, or 1GB for ~10 running bots

running bot means 1 currency pair?

Exactly. But since version 3.2 needs ~100MB RAM at start, I'll reduce that numbers:

Select the 512MB RAM option for ~3 running trade pairs, 1GB for ~8 pairs or 2GB for ~17 pairs

[NCick27]

Sorry if this is a poor question but I am new to everything.

Unsure of how to detach from the bot and do not want to accidentally close PuTTY and lose myself BTC
For example:
After I have typed "glog BTC-XRP" and it shows me the real time logs but how do I leave one log to check a different log.

Other questions:
How long does it take for the bot to start trading as the logs I have checked are empty?
Is there any way for me to integrate this into a webserver so I can monitor profit/ losses?
How can I change bot settings?

Thanks so much for any advice as I said I am very new.

[beer-k0in]

Hi,

Unsure of how to detach from the bot and do not want to accidentally close PuTTY and lose myself BTC
For example:
After I have typed "glog BTC-XRP" and it shows me the real time logs but how do I leave one log to check a different log.

I understand your concerns, but do not worry. The bot is running on your server and if you close putty, all you do is close your connection to the server, but not the server itself. It is like having your PC running and turning of your monitor - the PC still run's but you do not see it.
1. If you close putty, just start it again.
2. To get out of "glog BTC_XRP" press CTRL+C

How long does it take for the bot to start trading as the logs I have checked are empty?

Depending on the strategy, the strategy settings and most important, the market. If the prices on the market are not low enough, the GUNBOT will not buy.
Check the log an look for something like:

Code: [Select]

PriceToBuy 33.333 priceToSell --not set--
LP 44.0000000 > prBuy 33.333That means the GUNBOT wants to buy at 33.333 but the current price is to high (44.000)

Have patience.

Is there any way for me to integrate this into a webserver so I can monitor profit/ losses?

Sure, if you install a webserver and parse the logs and build the webapp to serve the parsed data.
But I think no one has build an easy to install webserver yet.

How can I change bot settings?

Open the 'poloniex-BTC_XXX-config.js' (the file will be empty) and add the settings you want to change.
I'm going to build a tool to do this like generating the configs, but that will take a while.

BK

[jung]

ERROR! - There was an error starting GUNBOT for BTC_EXP:
1


I have this error everytime I want to run it. Any Idea?

[beer-k0in]

ERROR! - There was an error starting GUNBOT for BTC_EXP:
1


I have this error everytime I want to run it. Any Idea?

1. How many trade pairs are you going to start?
2. If you enter "gl", is there an empty list or is BTC_EXP in the list, but stopped?
3. Is there a log file for BTC_EXP? (enter "tail -n 30 /opt/gunbot/poloniex-BTC_EXP-log.txt" to see the last 30 lines, if it exists)
4. Is there a config file for BTC_EXP? (enter "ll /opt/gunbot/*BTC_EXP*" to see all files for that pair)

[Hexvenn]

It's quite lovely! Especially the interface to add coins. Thank you.

It was relatively painless, but I had to restart the script a bit. The first time it didn't get all the dependencies and my apt-get got itself stuck, and I also had this "1" error. I rebooted the VPS and ran the script again and it went smoothly then. I am running AWS t2.micro so perhaps the initial configuration is a tad different than your provider - nevertheless, the script does very well!

[beer-k0in]

It's quite lovely! Especially the interface to add coins. Thank you.

Thanks. I plan to extend the 'gadd' interface to be able to change the strategy and strategy settings for each trade pair you want to add.

It was relatively painless, but I had to restart the script a bit. The first time it didn't get all the dependencies and my apt-get got itself stuck, and I also had this "1" error. I rebooted the VPS and ran the script again and it went smoothly then. I am running AWS t2.micro so perhaps the initial configuration is a tad different than your provider.

Interesting. I will try AWS later this week. Thanks for your feedback!

BK

[jung]

ERROR! - There was an error starting GUNBOT for BTC_EXP:
1


I have this error everytime I want to run it. Any Idea?

1. How many trade pairs are you going to start?
2. If you enter "gl", is there an empty list or is BTC_EXP in the list, but stopped?
3. Is there a log file for BTC_EXP? (enter "tail -n 30 /opt/gunbot/poloniex-BTC_EXP-log.txt" to see the last 30 lines, if it exists)
4. Is there a config file for BTC_EXP? (enter "ll /opt/gunbot/*BTC_EXP*" to see all files for that pair)

1. I tried multiple options from 1 pair to 6
2. empty, nothing change
3. nope
4. nope

And there is another problem with API. When I put api I recive 422 error or 403 but when I run it on windows with 3.1 version all of them works :|

[beer-k0in]

Could be an old version of the generator. Please check the version with

Code: [Select]

npm -g list generator-gunbot
If it is not generator-gunbot@0.1.8 please run

Code: [Select]

npm -g install generator-gunbot
and try again.

[Hexvenn]

How can I update to the new version from repo without messing things up?
Cheers

[beer-k0in]

How can I update to the new version from repo without messing things up?

1. Login to your server.
2. Run this update script

Code: [Select]

curl -qsL https://raw.githubusercontent.com/BeerK0in/generator-gunbot/master/update.sh | bash --3. Run again

Code: [Select]

ginit4. Press ENTER for every question (it will use your last settings)
5. Press ENTER on the conflict of ALLPAIRS (*)
6. Wait till all bots are started.
7. [optional] If you have changed some configs manually, please copy them from the backup folder back into the gunbot folder:

Code: [Select]

cp /opt/gunbot-backup-<NUMBER>/poloniex-BTC_XXX-config.js /opt/gunbot/
Done

Check with 'gl' if all bots are startet. If not, run this:

Code: [Select]

gstart BTC_XXX--------

(*)The conflict is because the default settings have changed a bit and it is very likely you see this question:

Code: [Select]

conflict ALLPAIRS-params.js
? Overwrite ALLPAIRS-params.js? (Ynaxdh)
BK

[cw]

This is very nice - my hat's off to you.  I decided to give this a try, even though I have my own servers, equipment and typically do everything by hand, and I'm a big fan of GNU screen, so didn't know what could be better than it.  Besides, why would I pay for something in the cloud for something I have "for free" at home?

However, I'm always looking to learn new things, and I've seen people talking about "pm2" and wondered what it was all about.  Plus, even though what I have at home is "free", $10/mo isn't too bad given what I make with gunbot, and there is the convenience factor of better 24x7 uptime.  Then again, on one server at home, I'm able to run 50 pairs and the box is still only at 5% CPU (against three exchanges)... so we'll have to see and compare later about the costs... but that's a different story.

So I went through the installation instructions and they were perfect.  I'm extremely impressed by your startup script.  Easy to use, easy to understand, and fast response.  Good job.

I now understand better what pm2 does - it obviates the need for screen.  While I see the benefits that has... I also see that I lose some control that screen gives me. For one, I no longer see the console output I had, and I see to lose the direct interaction with the STDIN/STDOUT (but I may just be missing that - or maybe that's what "glog" does....)

Once I realized how incredibly valuable your startup scripts were, I had to add in my multitail pieces.... since pm2 takes care of the "tee" and "screen" parts! (written up in this post: https://gunthy.org/index.php?topic=261.msg1164#msg1164)

So, if you install multitail:

Code: [Select]

root@gunny:/opt/gunbot# sudo apt install multitail
Reading package lists... Done
Building dependency tree
Reading state information... Done
The following NEW packages will be installed:
  multitail
0 upgraded, 1 newly installed, 0 to remove and 1 not upgraded.
Need to get 121 kB of archives.
After this operation, 322 kB of additional disk space will be used.
Get:1 http://mirrors.digitalocean.com/ubuntu xenial/universe amd64 multitail amd64 6.4.2-1build1 [121 kB]
Fetched 121 kB in 0s (1,258 kB/s)
Selecting previously unselected package multitail.
(Reading database ... 62956 files and directories currently installed.)
Preparing to unpack .../multitail_6.4.2-1build1_amd64.deb ...
Unpacking multitail (6.4.2-1build1) ...
Processing triggers for man-db (2.7.5-1) ...
Setting up multitail (6.4.2-1build1) ...

Create a file called "multitail.conf" in the /opt/gunbot directory:

Code: [Select]

#include:/etc/multitail.conf
#check_mail:0
colorscheme:gunbot
cs_re:red,magenta:< prBuy
cs_re:red,white:<= prBuy
cs_re:red,white:>= prSell
cs_re:red,yellow:> prSell
cs_re:red,yellow:Sell order placed
cs_re:red,yellow:.*ORDER INFORMATION.*
cs_re:red,yellow:.*profit.*
cs_re:white:.*MinPriceToSell.*
cs_re:white:.*MaxPriceToBuy.*
cs_re:white:.*MaxPriceToSell.*
cs_re:white:.*MinPriceToBuy.*
cs_re:yellow:^Direction:\s+\d+
cs_re:white,blue:Activating hot reconfig...
cs_re:white,red:INVALID LICENSE
cs_re:white,red:Rate limit exceeded
cs_re:white,red:Not enough BTC
cs_re:red:iInvalid nonce\!*
cs_re:red:ESOCKETTIMEDOUT\!*
cs_re:red:returned error\!*
cs_re:red:Error\!*
cs_re:red:.*failed.*
cs_re:red:.*no altcoins.*
cs_re:red:statusCode (.*)
cs_re:magenta:.*buy.*
cs_re:cyan:Open orders
cs_re:cyan:.*sell.*
cs_re:red:falls\syes
cs_re:green:grows\syes
cs_re:green:falls\sno
cs_re:yellow:Has order:.*
cs_re:yellow:.*have altcoins.*
cs_re:white:\:\:.*

Create a file called "stat.sh" in same directory:

Code: [Select]

#!/bin/sh
multitail -Z red,black,inverse --config multitail.conf -CS gunbot -x "%m %u@%h %f (%t) [%l]" -b 4 -s 2 -m 0 -n 100 \
   --mark-interval 200 \
        -Ev "before to buy" \
        -Ev "Cannot sell" \
        -Ev "price to buy" \
        -Ev "MaxPriceToBuy" \
        -Ev "Last price <=" \
        -Ev "Last price >=" \
        -Ev "Last price >" \
        -Ev "Last price >" \
        -Ev "price  grows \(" \
        -Ev "price  falls \(" \
        -Ev "^need " \
        -Ev "^Callback:" \
        -Ev "no altcoins" \
        -Ev "Altcoin Balance" \
        -Ev "\+\+\+"\
        -Ev "\[\]"\
        -Ev "-------" \
        -Ev "is too high" \
        -Ev "price is too low" \
        -Ev "price too high" \
        -Ev "waiting to sell" \
        -Ev "Creating market client" \
        -Ev "Loading config" \
        -Ev "Not supergun:" \
        -Ev "Supergun: " \
        -Ev "avCoins" \
        -Ev "CANCEL sell orders" \
        -Ev "CANCEL buy orders" \
        -Ev "SEll on start" \
        -Ev "^$" \
        -Ev "^\s+\d+\.\d+$" \
        -Ev "No order" \
        -Ev ":::" \
        -Ev "Error\!\!\! statusCode 422" \
        -Ev "Will repeat" \
        -Ev "cycle" \
        -Ev "No basePrice" \
        -Ev "need_" \
        -Ev "Response:" \
        -Ev "Bot responsed" \
        -Ev "next delay" \
        -Ev "^\*" \
        -Ev "\*\*" \
        /root/.pm2/logs/*-out*.log

Make the file executable:

Code: [Select]

root@gunny:/opt/gunbot# chmod +x ./stat.sh

and run the stats page...

Code: [Select]

./stat.sh

Then you can get a single page viewer of all the log files ... BUT FILTERED AND COLORED so you can watch them, but only the interesting parts.  I can't see them very easily right now because evidently I have an invalid license... so the screenshot isn't very useful.  Also, I'm composing this on my laptop right now, so the monitor isn't very large, so each window is a bit smaller than normal.  Typically you'd see a few more lines of each bot instance.



If there's a multitail-like console built-in to pm2, and I missed it, let me know.  If you read the link to my post above, I go into great detail about why I put together my multitail console and the ideas I have for the next one.

All in all, I have to say - great work!


Also, FYI - I started 16 pairs on a 2GB memory / 40GB / SFO2 Ubuntu 16.04.2 x64 unit and CPU is pegged at 100% - just wanted to pass that along to others.

[Pro]

This is almost exacly what I need
Let's talk details soon.
Thanks

[beer-k0in]

Hi cw,

I'm extremely impressed by your startup script.

wow, thanks. I'm really glad you like it.
I'm planing to change a few settings and extend the process of adding new pairs and just restarting some / all bots. Since I'm using GUNBOT a little longer now, my requirements have changed a bit since my first steps.  

Lack of output

Actually I'm also missing output in this setup. pm2 is a really solid process manager, but having 16 trade pairs running, "pm2 monit" (all logs of all processes in one window without colors) is not useful at all.
Your setup with all the log screen is too much for my current needs - too much information for my slow brain. So I'm building a monitor to show all information in on screen in one table. It is not ready to give it away yet (it does not collect all errors and some price infos are still ignored), but here is a sneak peek I hope to finish it next week.




CPU load

Can you please check again? I'm running the same droplet (2GB) with 16 pairs and my load is at 2% (Stepgain) to 4% (BB).




BK

[cw]

Dude.   I love your new monitor.  We should probably team up somehow on our ideas. Did you see the mock up I posted in my link of the gunbot console I'm working on from last month?  Mine parses the logs as well (but is dependent on v203 console logs which is why I use "tee"). It looks like pm2 fixes this problem for me so I'm super happy to have found your post.  I never would have dove into pm2 without being curious about your amazing generator-gunbot script. 

I think we're on the same track, don't you?

I want a per-pair latest snapshot of useful info.  I do NOT want a log I have to scan. Logs don't help me see what is current.  Logs help me trouble what happened in the past.  Most of the time I want to see a snapshot of the NOW with a hint of the past.  I like a lot of what you put in there.  I'd add something like trades: x/x/x and that would be 1hr/6hr/12hr/24hr/ etc so we could see

Also gnuplot and other Linux CLI plotting tools have some powerful capabilities where I wanted to have some easy ANSI one liner visual tools to show activity per pair for trades, prices,  directions, profit (relative to all pairs in table) and all that stuff. 

For a while I thought the json would have what I wanted but in previous versions Gunthar only saved it when he made a trade.  I needed it updated every cycle and I asked that as a feature.  We'll see.  Because if we get that info not only could we do a CLI interface but we could do a PHP web interface as well if we wanted.  (With buttons and an API to tweak settings).

 

[redindian]

Wow, this is great! Was running on Linux using screen (ouch!) but this is so much better. Thanks a lot!

[Pro]

I have some Ideas for that as well.
Could it also create a swap virtual RAM, so you could run 50 pairs on a 5$ Setup, instead of just 5.
This would be great