Linux Connection Throttler/SSL proxy for Poloniex

[dj31]

Disclaimer:

This may work for you, and it may not.   It's not a silver bullet, and you will still get 422's.   No 'optimizations' are performed.  All it does is globally limit the number of simultaneous connections to poloniex.    It currently has the polo IP hard coded, so if it changes -- you gotta change it.  It will redirect all local https requests to poloniex.

Requirements:
- 'nix server
- haproxy installed
- ability to edit hosts file
- port 443 available

Step 1:   Setup haproxy

This is a barebones, no logging, no frills config.

Code: [Select]

global
   maxconn 4096
   user haproxy
   group haproxy
   pidfile     /run/haproxy.pid
   daemon

defaults
   mode  http
   option   httplog
   option   dontlognull
   retries  3
   option redispatch
   maxconn  2000
   timeout connect 10000
   timeout server 50000
   timeout client 50000

# Host HA-Proxy web stats on Port 3306 (that will confuse those script kiddies)
listen HAProxy-Statistics
    bind :3306
    mode http
    option httplog
    option httpclose
    stats enable
    stats uri /haproxy?stats
    stats refresh 20s
    stats show-node
    stats show-legends
    stats show-desc Workaround haproxy for SSL
    stats auth admin:nimda

frontend ssl_relay
    bind :443
    mode tcp
    option tcplog
    option socket-stats
    maxconn  5
    default_backend ssl_polo

backend ssl_polo
   mode tcp
   option tcplog
   balance roundrobin
   hash-type consistent
   server x_polo 104.20.12.48:443

-  Change the stats auth <user>:<password> to your liking.
-  If port 3306 isn't available for stats, change it to whatever you like
-  Set maxconn to however many maximum simultaneous connections you want

You can view stats at:   http://<yourserver>:3306/haproxy?stats

Step 2.  Redirect polo requests

Edit your /etc/hosts file and append the following to the end

Code: [Select]

<haproxyIPAddress>   poloniex.com


I'm sure there are some optimizations/tweaks, but it's a starting point.

[AlfonseX]

Cool job
I didn't know haproxy, I will give it a try and dig though its configuration.
How do you find Polo IP?

Keep in touch.

[beer-k0in]

How do you find Polo IP?

Actually polo uses 2 IPs: 104.20.12.48 and 104.20.13.48. (Bittrex and Kraken using 5).

A tool to get IPs for a domain name is nslookup:

Code: [Select]

$ nslookup poloniex.com

Non-authoritative answer:
Name: poloniex.com
Address: 104.20.13.48
Name: poloniex.com
Address: 104.20.12.48


BK

[dj31]

Some slight tweaks.

#1 - use localhost instead of adapter's IP
#2 - use both polo servers on the backend.

New Config

Code: [Select]

global
   maxconn 4096
   user haproxy
   group haproxy
   pidfile     /run/haproxy.pid
   daemon

defaults
   mode  http
   option   httplog
   option   dontlognull
   retries  3
   option redispatch
   maxconn  2000
   timeout connect 10000
   timeout server 50000
   timeout client 50000

listen HAProxy-Statistics
    bind :3306
    mode http
    option httplog
    option httpclose
    stats enable
    stats uri /haproxy?stats
    stats refresh 20s
    stats show-node
    stats show-legends
    stats show-desc Workaround haproxy for SSL
    stats auth admin:nimda

frontend ssl_relay
    bind 127.0.0.1:443
    mode tcp
    option tcplog
    option socket-stats
    maxconn  3
    default_backend ssl_polo

backend ssl_polo
   mode tcp
   option tcplog
   balance roundrobin
   hash-type consistent
   server x_polo 104.20.12.48:443
   server x_polo2 104.20.13.48:443

New /etc/hosts entry

Code: [Select]

127.0.0.1       poloniex.com


[Seki92]

error

::::2017/06/25 16:59:10   Error Error: connect ECONNREFUSED 127.0.0.1:443

::::2017/06/25 16:59:11   Error Error: connect ECONNREFUSED 127.0.0.1:443

[AlfonseX]

Port 443 is not open, do :

Code: [Select]

sudo iptables -A INPUT -p tcp -m tcp --dport 7777 -j ACCEPT
See https://help.ubuntu.com/community/IptablesHowTo

[agis]

error

::::2017/06/25 16:59:10   Error Error: connect ECONNREFUSED 127.0.0.1:443

::::2017/06/25 16:59:11   Error Error: connect ECONNREFUSED 127.0.0.1:443

haproxy is doing what it's supposed to do: accepting 3 simultaneous connections and rejecting the rest (read the conf you just copied and pasted)

[AlfonseX]

Finally, I'm a noob in network concern, what is the purpose of that? I know load balancing and so on, but in this case I have difficulties to understand.

Thanks.

[allanster]

What they're trying to do is the same thing I'm doing in Windows, we're trying to limit the number of connections from Gunbot to Poloniex. I just spent hours the other night testing different angles of attacking this same problem. Registry hacks, 3rd party apps, etc...

I can't speak to what their purpose is but I suppose it's the same as mine, to reduce 422 timeout errors.

I also want to prevent exceeding the 6 api calls per second rule. Problem is, I don't know how many calls are made per connection or session. So even if I limit it to one connection, I may still get 429 error.

May I ask why I see everyone limiting to 3 connections instead of 6? Has someone already figured this out?

[allanster]

Solved! (free) -> https://gunthy.org/index.php?topic=570.msg2966#msg2966