CSV|JSON structure

* * *

POTREX DATA FORMAT - HOMEPAGE

Each entry represent a proposed video from Pornhub.
They are video snippet you might click on while visiting the platform.

      {
        "title": "Sunny Sextape on the Sofa! Squirt, deepthroat",
        "authorName": "Leolulu",
        "authorLink": "/pornstar/leolulu",
        "duration": "17:15",
        "href": "/view_video.php?viewkey=ph5e18b11299830",
        "savingTime": "2020-01-19T22:18:10.522Z",
        "metadataId": "738c411c67c7b6107bbb3ff8631070011a814f48",
        "clientTime": "2020-01-19T22:17:48.000Z",
        "size": 421227,
        "randomUUID": "INITucmr5condtj2zkfy9o6cv4",
        "selector": "body",
        "incremental": 0,
        "amountGrossDimension": 0,
        "packet": 0,
        "type": "home",
        "processed": true,
        "step": 0,
        "session": 1,
        "pseudo": "blueberry-cake-pistachio",
        "sectionName": "Hot Porn Videos In United States",
        "sectionHref": "/video?o=ht&cc=us",
        "sectionOrder": 0,
        "displayOrder": 0,
      },

PUPPEETER 101

* * *

Chrome(ium) automation through dev tools!

It's a node library with JS extensions

Made with UI testing and browser automation in mind

Full browser configuration capabilities

From its github repository

      

      const puppeteer = require('puppeteer');
      (async () => {
        const browser = await puppeteer.launch();
        const page = await browser.newPage();
        await page.goto('https://example.com');
        await page.screenshot({ path: 'example.png' });
        await browser.close();
      })();
      

AN EXPERIMENT SAMPLE

* * *

In general for an experiment we mean a set of commands which automates the navigation of our bot.

This includes a landing page, a list of videos to be watched and the time to be spent on each page.

From the potrex repository in methodology/json/phase1/guardoni2.json

[{
    "name": "Homepage first",
    "url": "https://www.pornhub.com/",
    "loadFor": "17s",
    "screenshotAfter": "5s"
},{
    "name": "Hentai-1",
    "loadFor": "3m",
    "screenshotAfter": "5s",
    "url": "https://www.pornhub.com/view_video.php?viewkey=ph604a68bb36307"
},{
    "name": "Hentai-2",
    "loadFor": "3m",
    "url": "https://www.pornhub.com/view_video.php?viewkey=ph604a625b562ea"
},{
...
      

CODE SAMPLES

* * *

From methodology/src/domainSpecific.js
you can see that we save everything from localstorage

for (let i = 0; i < localStorage.length; i++) {
  const key = localStorage.key(i);
  json[key] = localStorage.getItem(key);
...
const cookies = await page._client.send('Network.getAllCookies');
      

CODE SAMPLES

* * *

...Or how third party trackers are saved

const up = url.parse(reqpptr.url());
if(_.endsWith(up.host, 'pornhub.com') ||
_.endsWith(up.host, 'phncdn.com') )
  return;
const full3rdparty = {
  method: reqpptr.method(),
  host: up.host,
  pathname: up.pathname,
...
if(full3rdparty.method != ‘GET’)
full3rdparty.postData = reqpptr.postData();

In the European context some precious tools are already filling up the void

* * *

General Data Protecion Regulation (since 2018)

Privacy and Electronic Commerce Directive (since 2002)

ecommerce directive (since 2000)

… and more