Mimers brunn

Mimersbrunn.se is a website where students can upload self written essays for others to read, comment on and rate.

The problem lies in the way that the article read count is updated, as the server is simply counting the amount of requests made for a specific article, rather than filtering requests by IP-address or browser fingerprint. This enables users to send a large amount of requests for an article that will all go towards the view count. In my case I wanted to make the article about Danderyds Gymnasium the most viewed article on the website.

Tools used

Python for sending requests
Node.js for creating the request sending server
glitch.com for hosting the node.js server

Method

To make an article the most viewed article, it is good to know how many views the current most read article has. This can be done by iterating through all ~60.000 article IDs in the URL, which is just what this script does:

Python

Copy all

Select all

Open in repl.it

from time import sleep
from tqdm import trange
import requests
from bs4 import BeautifulSoup
import urllib3
import json
from threading import Thread

# Disable warnings for insecure requests
urllib3.disable_warnings(urllib3.exceptions.InsecureRequestWarning)


def article_data(articleID):
    global ARTICLES
    try:
        url = f"https://mimersbrunn.se/article?id={articleID}"
        payload = {}
        headers = {}

        # Send the request to the remote server
        response = requests.request(
            "GET", url, headers=headers, data=payload, verify=False)

        # Parse the response
        soup = BeautifulSoup(
            response.content, 'html.parser', from_encoding='utf-8')

        # Read the article views and title
        views = soup.findAll('small')[0].findAll('span')[0].decode_contents()
        title = soup.findAll('title')[0].decode_contents()

        # Add data to main list, so that it can be read later
        ARTICLES.append({'Views': int(views), 'Title': title, 'ID': articleID})
    except Exception as e:
        # If server is down or article does not exist
        ARTICLES.append(
            {'Views': -1, 'Title': 'Unable to read data', 'ID': articleID})


def _thread_article_data(startID, stopID):
    for ID in range(startID, stopID):
        article_data(ID)


# User preferences
MAX_ARTICLES = 60000
NUM_THREADS = 6

# Global variable definition
ARTICLES = []
THREADS = []
STEP = MAX_ARTICLES//NUM_THREADS

# Create threads
for i in range(NUM_THREADS):
    THREADS.append(Thread(target=_thread_article_data,
                          args=(i*STEP, (i+1)*STEP)))

# Start threads
for t in THREADS:
    t.start()

# Display progress
for i in trange(MAX_ARTICLES):
    for _ in range(20):
        if i < len(ARTICLES):
            break
        sleep(0.5)
    else:
        break

# Sort the list by views
ARTICLES = sorted(ARTICLES, key=lambda i: i['Views'], reverse=True)

# Write all data to "articleData.json" so we can look through it later
with open('articleData.json', 'w') as f:
    f.write(json.dumps(ARTICLES))

It turns out the most viewed article is the Islänningasagor article with 2,356,325 views at the time of writing.

In order to get the Danderyds Gymnasium article 2M views ahead I had to write another script to automate the website requests.
Here is the first script I wrote:

Python

Copy all

Select all

Open in repl.it

from time import sleep
from tqdm import trange
import requests
from bs4 import BeautifulSoup
import urllib3
import json
from threading import Thread
urllib3.disable_warnings(urllib3.exceptions.InsecureRequestWarning)


def generate_view():
    url = "https://mimersbrunn.se/article?id="+str(ARTICLE_ID)

    payload = {}
    headers = {}

    response = requests.request(
        "GET", url, headers=headers, data=payload, verify=False)


def _thread_view_generator(views):
    global VIEW_COUNT
    for _ in range(views):
        while 1:
            try:
                generate_view()
                break
            except:
                sleep(0.1)
        VIEW_COUNT += 1


ARTICLE_ID = 1441
VIEW_COUNT = 0
THREADS = []
NUM_VIEWS = 2000000
NUM_THREADS = 5
STEP = NUM_VIEWS//NUM_THREADS

for i in range(NUM_THREADS):
    THREADS.append(Thread(target=_thread_view_generator,
                          args=(STEP,)))

for t in THREADS:
    t.start()

for i in trange(NUM_VIEWS):
    for _ in range(20):
        if i < VIEW_COUNT:
            break
        sleep(0.5)
    else:
        break

Although this method works, it only gets about 500,000 views per night. This is a bit slower than I would hope, as I prefer not having my computer fan running the whole night. A better solution was to translate the python script to Node.js and host it externally from glitch.com with a visual interface of stats.

Node.js

Copy all

Select all

Open in repl.it

const https = require("https");
var restify = require("restify");
var fs = require("fs");

var SOURCE = fs.readFileSync("index.html", "utf8");
var NUM_REQUESTS = 0;
var ERRORS = 0;
var WAITING_TIME = 0;
var UPTIME = 0;
var SITE_DOWN = false;
var REQUESTS_PER_SECOND = 0;

// Setup Restify Server
var server = restify.createServer();
server.listen(process.env.PORT || 3000, function () {
  console.log("Listening to %s", process.env.PORT);
});

// Prints a message on the homepage for visitors
server.get("/", function (err, req) {
  let TOTAL_UPTIME = format(process.uptime());
  let WEBSITE_NAME = "mimersbrunn.se";
  let TOTAL_PINGS = NUM_REQUESTS;
  let PING_SPEED = REQUESTS_PER_SECOND + " pings/s";

  if (SITE_DOWN) {
    PING_SPEED =
      "Site down for " + (WAITING_TIME / 3).toString(10).split(".")[0] + "s";
  }

  req.write(
    SOURCE.replace("{WEBSITE_NAME}", WEBSITE_NAME)
      .replace("{TOTAL_UPTIME}", TOTAL_UPTIME)
      .replace("{TOTAL_PINGS}", TOTAL_PINGS)
      .replace("{PING_SPEED}", PING_SPEED)
  );
  req.end();
});

server.get("/data", function (err, req) {
  let TOTAL_UPTIME = format(process.uptime());
  let WEBSITE_NAME =
    'mimersbrunn.se';
  let TOTAL_PINGS = NUM_REQUESTS;
  let PING_SPEED = REQUESTS_PER_SECOND + " pings/s";

  if (SITE_DOWN) {
    PING_SPEED =
      "Site down for " + (WAITING_TIME / 3).toString(10).split(".")[0] + "s";
  }

  req.write(
    WEBSITE_NAME.toString(10) +
      "|" +
      TOTAL_UPTIME.toString(10) +
      "|" +
      PING_SPEED.toString(10) +
      "|" +
      TOTAL_PINGS.toString(10)
  );
  req.end();
});

server.get(/^\/?^((?!data).)*$/, function (err, req) {
  let TOTAL_UPTIME = format(process.uptime());
  let WEBSITE_NAME = "mimersbrunn.se";
  let TOTAL_PINGS = NUM_REQUESTS;
  let PING_SPEED = REQUESTS_PER_SECOND + " pings/s";

  if (SITE_DOWN) {
    PING_SPEED =
      "Site down for " + (WAITING_TIME / 3).toString(10).split(".")[0] + "s";
  }

  req.write(
    SOURCE.replace("{WEBSITE_NAME}", WEBSITE_NAME)
      .replace("{TOTAL_UPTIME}", TOTAL_UPTIME)
      .replace("{TOTAL_PINGS}", TOTAL_PINGS)
      .replace("{PING_SPEED}", PING_SPEED)
  );
  req.end();
});

const websiteURL = "https://mimersbrunn.se/article?id=1441";

function sleep(ms) {
  return new Promise((resolve) => {
    setTimeout(resolve, ms);
  });
}

function format(seconds) {
  function pad(s) {
    return (s < 10 ? "0" : "") + s;
  }
  var hours = Math.floor(seconds / (60 * 60));
  var minutes = Math.floor((seconds % (60 * 60)) / 60);
  var seconds = Math.floor(seconds % 60);

  return pad(hours) + ":" + pad(minutes) + ":" + pad(seconds);
}

async function main() {
  let proceed = false;
  while (true) {
    https
      .get(websiteURL, (resp) => {
        let data = "";

        // A chunk of data has been received.
        resp.on("data", (chunk) => {
          data += chunk;
        });
        resp.on("end", () => {
          NUM_REQUESTS += 1;
          proceed = true;
        });
      })
      .on("error", (err) => {
        console.log("Error: " + err.message);
        ERRORS += 1;
        proceed = true;
      });

    while (!proceed) {
      await sleep(200);
      WAITING_TIME += 0.2;
      if (WAITING_TIME > 30 && SITE_DOWN === false) {
        SITE_DOWN = true;
      }
    }
    WAITING_TIME = 0;
    SITE_DOWN = false;
    proceed = false;
  }
}

async function reqCounter() {
  let oldRequestCount;
  let tempReqPerSec;
  let requestTimings = [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0];

  while (true) {
    oldRequestCount = NUM_REQUESTS;
    await sleep(500);
    tempReqPerSec = (NUM_REQUESTS - oldRequestCount) / 0.5;
    requestTimings.push(tempReqPerSec);
    requestTimings.shift();
    REQUESTS_PER_SECOND = requestTimings.reduce((a, b) => a + b, 0) / 10;
  }
}

// Start all functions
Promise.all([main(), main(), main(), reqCounter()]);

The completed website can be found here.

After around a week the Danderyds Gymnasium article finally became the most viewed article, once and for all concluding my project.

Result

I learned a lot about external hosting and the security of view counters during this project. I also learned more about working with node.js which made me make an api for my websites view counting.

Here are the files:

File size: 3kb

view_counter.py

File size: 2kb

view_generator.py

File size: 8kb

node_server.zip

Mimers brunn

Summary

Tools used

Method

Result