Gunthy

GUNBOT: The automatic profit generator => Technical Support & Development => Topic started by: dj31 on June 25, 2017, 02:42:26 AM

Title: Linux Connection Throttler/SSL proxy for Poloniex
Post by: dj31 on June 25, 2017, 02:42:26 AM: Disclaimer:

This may work for you, and it may not. It's not a silver bullet, and you will still get 422's. No 'optimizations' are performed. All it does is globally limit the number of simultaneous connections to poloniex. It currently has the polo IP hard coded, so if it changes -- you gotta change it. It will redirect all local https requests to poloniex.

Requirements:
- 'nix server
- haproxy installed
- ability to edit hosts file
- port 443 available

Step 1: Setup haproxy

This is a barebones, no logging, no frills config.
Code: [Select]
global maxconn 4096 user haproxy group haproxy pidfile /run/haproxy.pid daemon defaults mode http option httplog option dontlognull retries 3 option redispatch maxconn 2000 timeout connect 10000 timeout server 50000 timeout client 50000 # Host HA-Proxy web stats on Port 3306 (that will confuse those script kiddies) listen HAProxy-Statistics bind :3306 mode http option httplog option httpclose stats enable stats uri /haproxy?stats stats refresh 20s stats show-node stats show-legends stats show-desc Workaround haproxy for SSL stats auth admin:nimda frontend ssl_relay bind :443 mode tcp option tcplog option socket-stats maxconn 5 default_backend ssl_polo backend ssl_polo mode tcp option tcplog balance roundrobin hash-type consistent server x_polo 104.20.12.48:443
- Change the stats auth <user>:<password> to your liking.
- If port 3306 isn't available for stats, change it to whatever you like
- Set maxconn to however many maximum simultaneous connections you want

You can view stats at: http://<yourserver>:3306/haproxy?stats

Step 2. Redirect polo requests

Edit your /etc/hosts file and append the following to the end

Code: [Select]
<haproxyIPAddress> poloniex.com

I'm sure there are some optimizations/tweaks, but it's a starting point.
Title: Re: Linux Connection Throttler/SSL proxy for Poloniex
Post by: AlfonseX on June 25, 2017, 08:41:12 AM: Cool job ;)
I didn't know haproxy, I will give it a try and dig though its configuration.
How do you find Polo IP?

Keep in touch.
Title: Re: Linux Connection Throttler/SSL proxy for Poloniex
Post by: beer-k0in on June 25, 2017, 09:38:44 AM: Quote from: AlfonseX on June 25, 2017, 08:41:12 AM
How do you find Polo IP?

Actually polo uses 2 IPs: 104.20.12.48 and 104.20.13.48. (Bittrex and Kraken using 5).

A tool to get IPs for a domain name is nslookup:

Code: [Select]
$ nslookup poloniex.com Non-authoritative answer: Name: poloniex.com Address: 104.20.13.48 Name: poloniex.com Address: 104.20.12.48

BK
Title: Re: Linux Connection Throttler/SSL proxy for Poloniex
Post by: dj31 on June 25, 2017, 01:16:57 PM: Some slight tweaks.

#1 - use localhost instead of adapter's IP
#2 - use both polo servers on the backend.

New Config
Code: [Select]
global maxconn 4096 user haproxy group haproxy pidfile /run/haproxy.pid daemon defaults mode http option httplog option dontlognull retries 3 option redispatch maxconn 2000 timeout connect 10000 timeout server 50000 timeout client 50000 listen HAProxy-Statistics bind :3306 mode http option httplog option httpclose stats enable stats uri /haproxy?stats stats refresh 20s stats show-node stats show-legends stats show-desc Workaround haproxy for SSL stats auth admin:nimda frontend ssl_relay bind 127.0.0.1:443 mode tcp option tcplog option socket-stats maxconn 3 default_backend ssl_polo backend ssl_polo mode tcp option tcplog balance roundrobin hash-type consistent server x_polo 104.20.12.48:443 server x_polo2 104.20.13.48:443
New /etc/hosts entry
Code: [Select]
127.0.0.1 poloniex.com
Title: Re: Linux Connection Throttler/SSL proxy for Poloniex
Post by: Seki92 on June 25, 2017, 04:07:30 PM: error

::::2017/06/25 16:59:10 Error Error: connect ECONNREFUSED 127.0.0.1:443

::::2017/06/25 16:59:11 Error Error: connect ECONNREFUSED 127.0.0.1:443
Title: Re: Linux Connection Throttler/SSL proxy for Poloniex
Post by: AlfonseX on June 25, 2017, 07:08:10 PM: Port 443 is not open, do :

Code: [Select]
sudo iptables -A INPUT -p tcp -m tcp --dport 7777 -j ACCEPT
See https://help.ubuntu.com/community/IptablesHowTo
Title: Re: Linux Connection Throttler/SSL proxy for Poloniex
Post by: agis on June 25, 2017, 07:12:05 PM: Quote from: Seki92 on June 25, 2017, 04:07:30 PM
error

::::2017/06/25 16:59:10 Error Error: connect ECONNREFUSED 127.0.0.1:443

::::2017/06/25 16:59:11 Error Error: connect ECONNREFUSED 127.0.0.1:443

haproxy is doing what it's supposed to do: accepting 3 simultaneous connections and rejecting the rest (read the conf you just copied and pasted)
Title: Re: Linux Connection Throttler/SSL proxy for Poloniex
Post by: AlfonseX on June 25, 2017, 07:34:23 PM: Finally, I'm a noob in network concern, what is the purpose of that? I know load balancing and so on, but in this case I have difficulties to understand.

Thanks.
Title: Re: Linux Connection Throttler/SSL proxy for Poloniex
Post by: allanster on July 03, 2017, 02:50:13 AM: What they're trying to do is the same thing I'm doing in Windows, we're trying to limit the number of connections from Gunbot to Poloniex. I just spent hours the other night testing different angles of attacking this same problem. Registry hacks, 3rd party apps, etc...

I can't speak to what their purpose is but I suppose it's the same as mine, to reduce 422 timeout errors.

I also want to prevent exceeding the 6 api calls per second rule. Problem is, I don't know how many calls are made per connection or session. So even if I limit it to one connection, I may still get 429 error.

May I ask why I see everyone limiting to 3 connections instead of 6? Has someone already figured this out?
Title: Re: Linux Connection Throttler/SSL proxy for Poloniex
Post by: allanster on July 03, 2017, 06:02:08 PM: Solved! (free) -> https://gunthy.org/index.php?topic=570.msg2966#msg2966