Username: Password:

Author Topic: Linux proxy for 422 errors.  (Read 19736 times)

Offline kevinlekiller

  • Rookie
  • *
  • Posts: 8
    • View Profile
Linux proxy for 422 errors.
« on: June 27, 2017, 02:32:32 PM »
Hi, I've written a free and open source proxy for linux to solve the 422 errors. Hope it helps you.

https://github.com/kevinlekiller/polproxy

Grimdin

  • Guest
Re: Linux proxy for 422 errors.
« Reply #1 on: June 27, 2017, 05:02:23 PM »
Nice. I'm testing it now.

Seems to work. Great work. Not seeing any 422 errors.

It does use lot's of CPU, that would be nice to get down little bit.

Offline AlfonseX

  • Contributor
  • **
  • Posts: 101
  • Trop de chefs, pas assez d'indiens !
    • View Profile
Re: Linux proxy for 422 errors.
« Reply #2 on: June 27, 2017, 05:14:20 PM »
Wow, great job Kevin  :)
I will test it as soon as possible.

Just a question, in the README you say:
Quote
Create a new poloniex api key: https://poloniex.com/apiKeys

Why? Can't use an existing API Key? I'm not sure.
If you think I helped you, give me a drink:
in btc: 12aeQSpytxoehCEptQE8tUJVVSAS42LvXo
in eth: 0x02a611f0c15bccdb6fa8e5e4b0692ff6d77852bd

Grimdin

  • Guest
Re: Linux proxy for 422 errors.
« Reply #3 on: June 27, 2017, 05:28:15 PM »
Wow, great job Kevin  :)
I will test it as soon as possible.

Just a question, in the README you say:
Quote
Create a new poloniex api key: https://poloniex.com/apiKeys

Why? Can't use an existing API Key? I'm not sure.

I am using it using existing API key, works for me.

Offline AlfonseX

  • Contributor
  • **
  • Posts: 101
  • Trop de chefs, pas assez d'indiens !
    • View Profile
Re: Linux proxy for 422 errors.
« Reply #4 on: June 27, 2017, 05:31:05 PM »
Encouter this error:
Code: [Select]
Traceback (most recent call last):
  File "polproxy.py", line 5, in <module>
    from urllib.parse import urlencode
ImportError: No module named parse

Ok, I made:
Code: [Select]
sudo python3 polproxy.pythat solve this error.

But now GB fired me:
Code: [Select]
Creating market client for BTC_BCN pair
Activating hot reconfig...
###[ --2017/06/27 18:34:28--poloniex--BTC_BCN  --  Gunbot v3.3 cycle #1 --------------- ]
::::: collecting market data...
Server running at http://127.0.0.1:39873
(node:5606) DeprecationWarning: Calling an asynchronous function without callback is deprecated.
/snapshot/Users/Enrico/Desktop/xBot_v2.9.3/node_modules/request/request.js:188
      self._callback.apply(self, arguments)
                     ^

TypeError: Cannot read property 'statusCode' of undefined
    at Request._callback (evalmachine.<anonymous>:0:0)
    at self.callback (/snapshot/Users/Enrico/Desktop/xBot_v2.9.3/node_modules/request/request.js:188:22)
    at emitOne (events.js:96:13)
    at Request.emit (events.js:189:7)
    at Request.onRequestError (/snapshot/Users/Enrico/Desktop/xBot_v2.9.3/node_modules/request/request.js:884:8)
    at emitOne (events.js:96:13)
    at ClientRequest.emit (events.js:189:7)
    at TLSSocket.socketErrorListener (_http_client.js:358:9)
    at emitOne (events.js:96:13)
    at TLSSocket.emit (events.js:189:7)
    at connectErrorNT (net.js:1025:8)
    at _combinedTickCallback (internal/process/next_tick.js:74:11)
    at process._tickCallback (internal/process/next_tick.js:98:9)

$
« Last Edit: June 27, 2017, 05:42:08 PM by AlfonseX »
If you think I helped you, give me a drink:
in btc: 12aeQSpytxoehCEptQE8tUJVVSAS42LvXo
in eth: 0x02a611f0c15bccdb6fa8e5e4b0692ff6d77852bd

Offline kevinlekiller

  • Rookie
  • *
  • Posts: 8
    • View Profile
Re: Linux proxy for 422 errors.
« Reply #5 on: June 27, 2017, 05:44:55 PM »
Nice. I'm testing it now.

Seems to work. Great work. Not seeing any 422 errors.

It does use lot's of CPU, that would be nice to get down little bit.

Yeah, it needs some optimization, I have some ideas in mind, will test over the next few days.

Offline kevinlekiller

  • Rookie
  • *
  • Posts: 8
    • View Profile
Re: Linux proxy for 422 errors.
« Reply #6 on: June 27, 2017, 05:45:24 PM »
Wow, great job Kevin  :)
I will test it as soon as possible.

Just a question, in the README you say:
Quote
Create a new poloniex api key: https://poloniex.com/apiKeys

Why? Can't use an existing API Key? I'm not sure.

When you make a new key, the nonce starts at 0, but it will still work with an existing key.

Offline kevinlekiller

  • Rookie
  • *
  • Posts: 8
    • View Profile
Re: Linux proxy for 422 errors.
« Reply #7 on: June 27, 2017, 05:46:30 PM »
Encouter this error:
Code: [Select]
Traceback (most recent call last):
  File "polproxy.py", line 5, in <module>
    from urllib.parse import urlencode
ImportError: No module named parse

Ok, I made:
Code: [Select]
sudo python3 polproxy.pythat solve this error.

But now GB fired me:
Code: [Select]
Creating market client for BTC_BCN pair
Activating hot reconfig...
###[ --2017/06/27 18:34:28--poloniex--BTC_BCN  --  Gunbot v3.3 cycle #1 --------------- ]
::::: collecting market data...
Server running at http://127.0.0.1:39873
(node:5606) DeprecationWarning: Calling an asynchronous function without callback is deprecated.
/snapshot/Users/Enrico/Desktop/xBot_v2.9.3/node_modules/request/request.js:188
      self._callback.apply(self, arguments)
                     ^

TypeError: Cannot read property 'statusCode' of undefined
    at Request._callback (evalmachine.<anonymous>:0:0)
    at self.callback (/snapshot/Users/Enrico/Desktop/xBot_v2.9.3/node_modules/request/request.js:188:22)
    at emitOne (events.js:96:13)
    at Request.emit (events.js:189:7)
    at Request.onRequestError (/snapshot/Users/Enrico/Desktop/xBot_v2.9.3/node_modules/request/request.js:884:8)
    at emitOne (events.js:96:13)
    at ClientRequest.emit (events.js:189:7)
    at TLSSocket.socketErrorListener (_http_client.js:358:9)
    at emitOne (events.js:96:13)
    at TLSSocket.emit (events.js:189:7)
    at connectErrorNT (net.js:1025:8)
    at _combinedTickCallback (internal/process/next_tick.js:74:11)
    at process._tickCallback (internal/process/next_tick.js:98:9)

$

I'm not sure, not encountered that issue, have to head out, but can help you out later this evening.

Offline citedjade

  • Rookie
  • *
  • Posts: 3
    • View Profile
Re: Linux proxy for 422 errors.
« Reply #8 on: June 27, 2017, 07:13:50 PM »
thanks! facing a problem when using it with mac.
terminal always stay at "Updating polo ip".
seems missing some configs in my mac. can write a guide for mac-user?

Offline kevinlekiller

  • Rookie
  • *
  • Posts: 8
    • View Profile
Re: Linux proxy for 422 errors.
« Reply #9 on: June 27, 2017, 07:15:33 PM »
Encouter this error:
Code: [Select]
Traceback (most recent call last):
  File "polproxy.py", line 5, in <module>
    from urllib.parse import urlencode
ImportError: No module named parse

Ok, I made:
Code: [Select]
sudo python3 polproxy.pythat solve this error.

But now GB fired me:
Code: [Select]
Creating market client for BTC_BCN pair
Activating hot reconfig...
###[ --2017/06/27 18:34:28--poloniex--BTC_BCN  --  Gunbot v3.3 cycle #1 --------------- ]
::::: collecting market data...
Server running at http://127.0.0.1:39873
(node:5606) DeprecationWarning: Calling an asynchronous function without callback is deprecated.
/snapshot/Users/Enrico/Desktop/xBot_v2.9.3/node_modules/request/request.js:188
      self._callback.apply(self, arguments)
                     ^

TypeError: Cannot read property 'statusCode' of undefined
    at Request._callback (evalmachine.<anonymous>:0:0)
    at self.callback (/snapshot/Users/Enrico/Desktop/xBot_v2.9.3/node_modules/request/request.js:188:22)
    at emitOne (events.js:96:13)
    at Request.emit (events.js:189:7)
    at Request.onRequestError (/snapshot/Users/Enrico/Desktop/xBot_v2.9.3/node_modules/request/request.js:884:8)
    at emitOne (events.js:96:13)
    at ClientRequest.emit (events.js:189:7)
    at TLSSocket.socketErrorListener (_http_client.js:358:9)
    at emitOne (events.js:96:13)
    at TLSSocket.emit (events.js:189:7)
    at connectErrorNT (net.js:1025:8)
    at _combinedTickCallback (internal/process/next_tick.js:74:11)
    at process._tickCallback (internal/process/next_tick.js:98:9)

$

Ok, I'm back, do you see anything in the polproxy.py window ? Any errors there?

Offline kevinlekiller

  • Rookie
  • *
  • Posts: 8
    • View Profile
Re: Linux proxy for 422 errors.
« Reply #10 on: June 27, 2017, 07:24:05 PM »
thanks! facing a problem when using it with mac.
terminal always stay at "Updating polo ip".
seems missing some configs in my mac. can write a guide for mac-user?

Hi, unfortunately I don't have access to a Mac.

The "Updating polo ip." tries to get the poloniex IP using dig then pipes it through head to get just the IP address,

If you run the command in a terminal do you get the IP ? dig @8.8.8.8 +short poloniex.com | head -1

Offline citedjade

  • Rookie
  • *
  • Posts: 3
    • View Profile
Re: Linux proxy for 422 errors.
« Reply #11 on: June 27, 2017, 07:34:17 PM »
thanks! facing a problem when using it with mac.
terminal always stay at "Updating polo ip".
seems missing some configs in my mac. can write a guide for mac-user?

Hi, unfortunately I don't have access to a Mac.

The "Updating polo ip." tries to get the poloniex IP using dig then pipes it through head to get just the IP address,

If you run the command in a terminal do you get the IP ? dig @8.8.8.8 +short poloniex.com | head -1

oh... it return 104.20.13.48.
should it return 127.0.0.1?  seems that my dnsmasq is not working

Offline kevinlekiller

  • Rookie
  • *
  • Posts: 8
    • View Profile
Re: Linux proxy for 422 errors.
« Reply #12 on: June 27, 2017, 07:37:48 PM »
thanks! facing a problem when using it with mac.
terminal always stay at "Updating polo ip".
seems missing some configs in my mac. can write a guide for mac-user?

Hi, unfortunately I don't have access to a Mac.

The "Updating polo ip." tries to get the poloniex IP using dig then pipes it through head to get just the IP address,

If you run the command in a terminal do you get the IP ? dig @8.8.8.8 +short poloniex.com | head -1

oh... it return 104.20.13.48.
should it return 127.0.0.1?  seems that my dnsmasq is not working

It should return 104.20.13.48. Basically, we are telling dig to ask 8.8.8.8 (google) what the poloniex IP is.
You can use dig +short poloniex.com to check if dnsmasq is working.

If you want to see what DNS server was used :
dig poloniex.com | grep -i server
« Last Edit: June 27, 2017, 07:41:39 PM by kevinlekiller »

Offline kevinlekiller

  • Rookie
  • *
  • Posts: 8
    • View Profile
Re: Linux proxy for 422 errors.
« Reply #13 on: June 27, 2017, 09:20:01 PM »
Nice. I'm testing it now.

Seems to work. Great work. Not seeing any 422 errors.

It does use lot's of CPU, that would be nice to get down little bit.

Cpu usage should be better now.

Offline Seki92

  • Rookie
  • *
  • Posts: 7
    • View Profile
Re: Linux proxy for 422 errors.
« Reply #14 on: June 27, 2017, 10:49:54 PM »
how can i dow  :'( :'( :'(

Traceback (most recent call last):
  File "polproxy.py", line 3, in <module>
    import hmac, hashlib, os.path, pycurl, re, socket, ssl, subprocess, sys, threading, time, yaml
ImportError: No module named 'yaml'

Offline botreminer0011

  • Rookie
  • *
  • Posts: 15
    • View Profile
Re: Linux proxy for 422 errors.
« Reply #15 on: June 27, 2017, 10:54:53 PM »
how can i dow  :'( :'( :'(

Traceback (most recent call last):
  File "polproxy.py", line 3, in <module>
    import hmac, hashlib, os.path, pycurl, re, socket, ssl, subprocess, sys, threading, time, yaml
ImportError: No module named 'yaml'


Install dnsmasq / dig (dnsutils in arch linux) / python 3 / pycurl (python-pycurl in arch, or use pip install pycurl on other distros) / pyyaml (python-yaml in arch or pip install pyyaml other distros)


Offline Seki92

  • Rookie
  • *
  • Posts: 7
    • View Profile
Re: Linux proxy for 422 errors.
« Reply #16 on: June 28, 2017, 01:04:49 AM »
dont working after this

Grimdin

  • Guest
Re: Linux proxy for 422 errors.
« Reply #17 on: June 28, 2017, 03:28:53 PM »
Nice. I'm testing it now.

Seems to work. Great work. Not seeing any 422 errors.

It does use lot's of CPU, that would be nice to get down little bit.

Cpu usage should be better now.

Yes it is, much better, thx!

Was running it over night and founded it crashed on morning, some sort of memory leak i assume.

I have now latest version (from github) running, did have problems starting it (nonce errors), but after re-start it managed to get past it.

Ok, now i am seeing "socket hang up" errors in gunbot, too low bot delay?

Offline AlfonseX

  • Contributor
  • **
  • Posts: 101
  • Trop de chefs, pas assez d'indiens !
    • View Profile
Re: Linux proxy for 422 errors.
« Reply #18 on: June 28, 2017, 05:07:34 PM »
dont working after this

Have you tried to run like:
Code: [Select]
sudo python3 polproxy.py
If you think I helped you, give me a drink:
in btc: 12aeQSpytxoehCEptQE8tUJVVSAS42LvXo
in eth: 0x02a611f0c15bccdb6fa8e5e4b0692ff6d77852bd

Offline Seki92

  • Rookie
  • *
  • Posts: 7
    • View Profile
Re: Linux proxy for 422 errors.
« Reply #19 on: June 28, 2017, 10:18:58 PM »
no look

Traceback (most recent call last):
  File "polproxy.py", line 3, in <module>
    import hmac, hashlib, os.path, pycurl, re, socket, ssl, subprocess, sys, threading, time, yaml
ImportError: No module named 'pycurl'

Offline Rakeau

  • Rookie
  • *
  • Posts: 2
    • View Profile
Re: Linux proxy for 422 errors.
« Reply #20 on: June 29, 2017, 05:28:10 AM »
Rather than going to all the effort to setup a DNS server, in order to 'override' the DNS lookup of Polo, why not just add the relevant domains into the /etc/hosts file? That will cause them to resolve to 127.0.0.1. And should you want to stop using the proxy, you just remove them out of that file again. By running dnsmasq and putting the IP address override in dnsmasq.conf you're basically doing the same thing - and if you want to disable the proxy you would have to undo the same changes in the config file.

Far less messing around, less resource overhead, and fewer dependencies to setup and install this way. The less things there are to break, the better.

Setup can be summarised as:

1. Setup the proxy config file settings.yml
2. Run polproxy.py
3. Add or remove 127.0.0.1 poloniex.com to /etc/hosts to use / not use proxy.

I'm running an AWS Ubuntu VPS. using python uses Python 2.x, so you need to change your commands for python3. To get it running:


Setup:

sudo apt-get install libgnutls28-dev
sudo apt-get install libcurl4-gnutls-dev
sudo apt install python-pip
sudo pip install pycurl
sudo pip install pyyaml
sudo pip install urllib
(Might not be necessary as per AlfonseX's post https://gunthy.org/index.php?topic=541.msg2781#msg2781)
sudo setcap CAP_NET_BIND_SERVICE=+eip $(readlink -f $(which python3))
wget https://github.com/kevinlekiller/polproxy/archive/master.zip -O ~/polproxy.zip
cd ~
unzip polproxy.zip
cd polproxy-master/
cp settings.yml.example settings.yml
nano settings.yml
(<-- input API details, ctrl+x then Y to save and close)

This assumes you'll put the polproxy folder in your home folder (~/).


Running the proxy:

cd ~/polproxy-master
screen -S proxy -dm python3 polproxy.py
sudo sh -c 'echo 127.0.0.1 poloniex.com >> /etc/hosts'


By using screen, we can have the proxy running in the background and leave the terminal free for other tasks.


Checking the proxy + Killing the proxy using screen (crash course):

screen -list -- shows active screens
screen -r proxy -- goes into the screen running the proxy, you can check its output for errors etc.
Ctrl + A then D -- detaches from the screen.
Ctrl + C -- Kills the proxy and terminates the screen (fix /etc/hosts first -- see below)
sudo nano /etc/hosts -- Edit the hosts file. Either comment out the lines (# prefix) or just remove them + save.

With that I can run the proxy.

Basically, to use the proxy, we run it, and then change our /etc/hosts file to point poloniex.com to 127.0.0.1. To stop using the proxy, we remove the poloniex entry from the hosts file. You don't even need to stop your bots.


However, once updating /etc/hosts , if I tail Gunbot's logs, I get:

Code: [Select]
2017/06/29 16:07:24     ::::: collecting market data...
2017/06/29 16:07:25     *** PLNX CALLBACK | calling EMA ***,unable to verify the first certificate
2017/06/29 16:07:25     2017/06/29 16:07:25 Error: Error: unable to verify the first certificate
2017/06/29 16:07:25     !!! Cycle 1413 failed. Will repeat in 60s

So it has an effect - but there seems to be an SSL issue. I imagine that is because I set it up using beer-k0in's Generator script + gmon, so I can't set the TLS_REJECT thing for nodejs... (I'm not sure if I'll continue using that but I digress).

If I don't run it via gstart / pm2 start and instead run it manually using the command provided (NODE_TLS_REJECT_UNAUTHORIZED=0 ./gunthy-linuxx64 BTC_PAIR poloniex), it seems to work, but I still get errors (leading to cycle failures), process duplication and high CPU usage. Unfortunately I don't think this bot really seems to solve anything.

Given there's a Windows Gunbot Proxy floating around as well, given these proxies seem popular right now, maybe it would be worth Gunthar giving us the option to use a proxy server or allow us to set the API URL / Domain in Gunbot's settings, rather than hard-coding it in .. this would greatly simplify things. Even if an update does resolve the 422 errors.

Also - the Poloniex API suggests to use "api.poloniex.com", so this may also need to be added, either to dnsmasq config or /etc/hosts. However, it really depends on which domain name Gunbot is trying to resolve internally.
« Last Edit: June 29, 2017, 09:55:10 AM by Rakeau »

Offline sakito

  • Rookie
  • *
  • Posts: 2
    • View Profile
Re: Linux proxy for 422 errors.
« Reply #21 on: July 03, 2017, 12:16:56 PM »
Hi, it's working the proxy?

Offline allanster

  • Contributor
  • **
  • Posts: 128
  • lux ex tenebris • deuro mea umbra absens
    • View Profile
Re: Linux proxy for 422 errors.
« Reply #22 on: July 03, 2017, 06:07:17 PM »
more human than human