/ python

Getting petition data from JSON

Ok firstly DO NOT JUST DO THIS FOR LOLS! We are querying a live database that is currently 502'ing (Bad Gateway "The server was acting as a gateway or proxy and received an invalid response from the upstream server") due to the sheer amount of people who are trying to sign an important petition!

So right now in the UK we are seeing lots of people signing a petition to revoke article 50 so that the UK can remain in the EU. If you haven't already, and are inclined to do so..SIGN IT!

Get to the point Les

Screenshot-from-2019-03-21-11-06-06-1
I wanted to know if there was an API for interacting with the live data, and while there is an API for general use, it turns out that there is a JSON feed at the bottom of the page!
/me tips hat to Gareth

/me tips hat to Ben
With code provided by Ben_Nuttall I was able to see the data in Python.

Link to Ben's code

import requests
from time import sleep

url = "https://petition.parliament.uk/petitions/241584.json"

def get_count():
    count = None
    while True:
        r = requests.get(url)
        if r:
            return r.json()['data']['attributes']['signature_count']
        else:
            print("Could not reach petition site")

c = get_count()
print("{:,} signatures".format(c))
while True:
    sleep(60)
    n = get_count()
    print("{:,} signatures, {:,} new signatures in last minute".format(n, n-c))
    c = n

So with that raw data you can create charts with plotly etc.

Update: Easier Revision!!

Alan Bell on Twitter just mentioned that we can go directly to the signature count data via this URL >> https://petition.parliament.uk/petitions/241584/count.json

import requests
from time import sleep

So I changed this URL

url = "https://petition.parliament.uk/petitions/241584.json"

To

url = "https://petition.parliament.uk/petitions/241584/count.json"

As it goes directly to the signature count!

def get_count():
    count = None
    while True:
        r = requests.get(url)
        if r:

Then I changed

            return r.json()['data']['attributes']['signature_count']

To

            return r.json()['signature_count']

Everything else was as before.

        else:
            print("Could not reach petition site")
c = get_count()
print("{:,} signatures".format(c))
while True:
    sleep(60)
    n = get_count()
    print("{:,} signatures, {:,} new signatures in last minute".format(n, n-c))
    c = n

BISH BASH BOSH!

Peek-2019-03-21-11-16
I wanted to see if I could use the data in a terminal, as you do!
So after a little Googling I found this post on how to use jq, a tool to read, write and filter JSON from the terminal.

I installed jq on my Ubuntu machine.

$ sudo apt install jq

And then I ran the following command

$ curl -s "https://petition.parliament.uk/petitions/241584.json" | jq '.data'.'attributes'.signature_count

curl is used to get the data from the URL where the JSON data is.
| is a pipe to send the output of the previous command to be the input of the next.
jq is used to read and filter the JSON data
'.data'.'attributes'.'signature_count' is used to drill down to the specific data that is nested inside the JSON.

How did I know that '.data'.'attributes'.'signature_count' was where the data was? Well in Ben's example he references ['data']['attributes']['signature_count'] as the location of the signature data. After quickly learning how jq worked with nested data (nested dictionaries in Python) I was able to whip this up.

Update: Shorter URL

In the update above we used the shorter URL to go directly to the signature_count data. And we can do this with the BASH code too.

So now the one lind command to get the data and pass it to jq is

$ data="$(curl -s "https://petition.parliament.uk/petitions/241584/count.json" | jq .'signature_count')"

Then use echo to print the contents of the variable.

echo $data

FUN!!!

BONUS!!

I thought it would be great to see the live count pop up on the screen...because we all love pop ups right?
In the terminal I created a variable called data and use that to store the output of the running the previous jq command to filter the data from JSON.

$ data=curl -s "https://petition.parliament.uk/petitions/241584.json" | jq '.data'.'attributes'.'signature_count' | notify-send

Then to send that to the notifications section in Ubuntu I used notify-send and then passed it the variable to display the data.

$ notify-send $data

Press Enter and you see the live signature count!

Screenshot-from-2019-03-21-12-19-37

Have fun and use this responsibly!